1
00:00:00,180 --> 00:00:05,160
And as a systems person kind of ramping up on this particular space, I learned there's

2
00:00:05,160 --> 00:00:09,440
like very interesting metrics that show up. Like at Amazon, we obsessed a lot about having

3
00:00:09,440 --> 00:00:13,800
the right metrics and spending a lot of time identifying the right metrics. And I found

4
00:00:13,800 --> 00:00:16,440
these two metrics to be very, very fascinating.

5
00:00:17,460 --> 00:00:21,900
One of them measures capability and the other one measures reliability. So there's pass

6
00:00:21,900 --> 00:00:28,100
at K and it means that if I run the task 10 times,

7
00:00:28,820 --> 00:00:34,160
at least one of those 10 times the agent will give me the right answer, okay? So now,

8
00:00:34,180 --> 00:00:39,340
why is this useful? Let's say I have an agent that costs 1 % of the frontier models,

9
00:00:39,820 --> 00:00:44,020
but I can really, really detect when I have the right answer. Then I can just run that

10
00:00:44,020 --> 00:00:48,560
agent 10 times and still come out ahead probabilistically, right? So that's where pass

11
00:00:48,560 --> 00:00:53,540
at K could be very, very useful because I can just hedge and get an answer.

12
00:00:54,980 --> 00:01:01,220
And this is like, can I do this at all, right? And there's gonna be like cheap agents

13
00:01:01,220 --> 00:01:05,800
that can, models that can just never do it. But if you can do it one out of 10 or whatever

14
00:01:05,800 --> 00:01:10,340
the K number is, that's a capability assessment.