1
00:00:00,180 --> 00:00:06,040
So I think you're absolutely right. If you make agents compete, the method by which you

2
00:00:06,040 --> 00:00:12,200
will tell which one is better is the evals bit. And you will never know who might have

3
00:00:12,200 --> 00:00:17,460
done it better or worse unless you run all of them. But if you run multiple, then you

4
00:00:17,460 --> 00:00:22,360
have to, you know, kind of assess their completion. That said, the big takeaway from

5
00:00:22,360 --> 00:00:26,820
this is not that I'm not telling everybody to go run their eval system. My big takeaway

6
00:00:26,820 --> 00:00:29,220
that I hope to convey is that be skeptical.

7
00:00:30,360 --> 00:00:35,720
What works as the best thing for a particular solution today may not be the best solution

8
00:00:35,720 --> 00:00:41,420
for that particular problem. And you see this, like if you look at open routers, traffic

9
00:00:41,420 --> 00:00:43,460
as well, like you'll see how traffic shifts

10
00:00:44,740 --> 00:00:49,340
between, you know, entropic and open AI models over time because people are just discovering

11
00:00:49,340 --> 00:00:53,960
that they're different. So remain skeptical, disconfirm your beliefs. Today, Codex might

12
00:00:53,960 --> 00:00:57,500
be really good at a particular task with Astra and tomorrow it might be the worst of

13
00:00:57,500 --> 00:01:02,820
the batch. As long as you can continue evolving your setup, it's probably fine.