1
00:00:00,180 --> 00:00:06,940
All right, so we talked about this rigorous eval system. And we'll end with this.

2
00:00:07,820 --> 00:00:14,240
When you are testing a system, you're not just testing the efficacy of the model. And

3
00:00:14,240 --> 00:00:18,720
there's a Databricks paper from a couple of months ago that actually showed that in many

4
00:00:18,720 --> 00:00:25,040
cases, the harness actually matters more than the model itself. And, but you gotta zoom

5
00:00:25,040 --> 00:00:27,720
back out even further. It all starts with the human.

6
00:00:28,780 --> 00:00:32,720
And the human can write a very bad prompt or they can write a good prompt that's like

7
00:00:32,720 --> 00:00:36,780
well organized, well thought out. Sometimes I turn on the reasoning tokens and it's like

8
00:00:36,780 --> 00:00:40,660
literally judging me. Like this guy can't make up his mind. I've literally seen that

9
00:00:40,660 --> 00:00:46,400
in the reasoning tokens. But like a good prompter with a good prompt is where it starts

10
00:00:46,400 --> 00:00:52,120
with the right context. So you're not kind of circling around. And then the harness makes

11
00:00:52,120 --> 00:00:56,580
a big difference in terms of how it uses that context. It's trying to, you as a human

12
00:00:56,580 --> 00:01:01,260
are trying to express something. The prompt is trying to capture that intent. The harness

13
00:01:01,260 --> 00:01:07,820
is trying to synthesize that intent into a compiled thing that goes to the model, which

14
00:01:07,820 --> 00:01:12,780
then is deciding well, how much do I wanna think about this? And how much do I wanna,

15
00:01:13,460 --> 00:01:18,760
how hard do I wanna think about it and so forth. And we are not testing models. We learned

16
00:01:18,760 --> 00:01:22,740
very quickly that we weren't testing frontier labs or set the other labs. You're testing

17
00:01:22,740 --> 00:01:24,900
the entire end to end setup.