1
00:00:00,180 --> 00:00:05,180
when you're evaluating the system, you're running the experiments over and over and over

2
00:00:05,180 --> 00:00:06,500
again. There's repetition.

3
00:00:07,460 --> 00:00:12,520
And oftentimes we found ourselves where we would see a very high degree of variance on

4
00:00:12,520 --> 00:00:18,460
whether a agent or whether a model was able to complete the job. And it's very tempting

5
00:00:18,460 --> 00:00:24,480
to just roll up all of the results in an aggregate and assign a score. But as you start

6
00:00:24,480 --> 00:00:25,300
diving deeper,

7
00:00:26,640 --> 00:00:31,160
like we were initially very uncharitable to some of the models where, it scored very,

8
00:00:31,160 --> 00:00:35,800
very poorly. And it turned out, well, it was because we were just running the test and

9
00:00:35,800 --> 00:00:41,280
we started getting throttled on the models. And unless you looked into the traces to

10
00:00:41,280 --> 00:00:47,880
see what actually went wrong and addressed that kind of artifact inside of your

11
00:00:47,880 --> 00:00:53,120
evaluation harness, you would never know. This is why I was like, when the strands age

12
00:00:53,120 --> 00:00:57,340
harness team was talking, they made a post on LinkedIn saying that they had to up their

13
00:00:57,340 --> 00:01:00,000
bedrock code out of this much. And I was like, I'm so jealous of you guys.

14
00:01:01,720 --> 00:01:09,060
But you have to attribute the variance. So if you get something that's one off, and understanding

15
00:01:09,060 --> 00:01:14,840
why it failed, sometimes it'll just give you a very easy black and white artifact. But

16
00:01:14,840 --> 00:01:19,520
other times it actually teaches you a little bit about how the models behave. And it

17
00:01:19,520 --> 00:01:24,420
gives you a little bit better mastery on how to steer them together. It builds a lot

18
00:01:24,420 --> 00:01:28,860
more mechanical empathy as you start diving into the variance between models and then

19
00:01:28,860 --> 00:01:30,600
within the models.