1
00:00:00,180 --> 00:00:05,540
Once you're at a job, why wouldn't you apply the same process to a coding agent just

2
00:00:05,540 --> 00:00:08,200
because it's driven by a large language model?

3
00:00:09,780 --> 00:00:14,460
So the way to think about coding evals and the importance of investing in them is so

4
00:00:14,460 --> 00:00:21,160
that you have data -driven approaches to measure and to act upon results that are gathered

5
00:00:21,160 --> 00:00:26,700
on your coding agents and how it is performing against your system, which by the way

6
00:00:26,700 --> 00:00:31,820
is constantly changing as well. Most of you have a continuous running set of tests, whether

7
00:00:31,820 --> 00:00:37,060
it's integration tests, performance tests, low tests that run on a regular basis. The

8
00:00:37,060 --> 00:00:41,480
same needs to be done for coding agents because guess what? The stack underneath you

9
00:00:41,480 --> 00:00:46,800
is changing all the time. Models are changing, getting updated. Harnesses are updating.

10
00:00:47,320 --> 00:00:51,840
Actually, I think something like clock code auto -updates now when you restart it. So

11
00:00:51,840 --> 00:00:55,240
sometimes you don't even get a choice. You can't lock everything down. You can't pin

12
00:00:55,240 --> 00:01:02,120
it. So that is why we bother, and that is how we started on our journey to write a

13
00:01:02,120 --> 00:01:07,060
rigorous eval for our coding agent and got ourselves into a whole can of worms.