WEBVTT

00:00:00.689 --> 00:00:07.329
Daniela and I spend a lot of time profiling high -performance systems. And that

00:00:07.329 --> 00:00:12.669
is actually quite fun and very, very easy because you know when something has been made

00:00:12.669 --> 00:00:13.169
faster.

00:00:14.549 --> 00:00:20.709
What we're gonna talk about here today is about evaluating coding agents.

00:00:21.709 --> 00:00:28.289
And that is not a very black and white area. And we've been spending

00:00:28.289 --> 00:00:34.569
a few months just diving down this rabbit hole of how do we build evals for coding agents.

00:00:35.089 --> 00:00:39.769
And every time, every day, we just get humbled by something that we did not think of.

00:00:39.949 --> 00:00:45.389
It's a genuinely hard problem. So we're not here to give you solutions to all of this.

00:00:45.449 --> 00:00:50.289
We're just gonna tell you about a bunch of stumbles that we have come across as we try

00:00:50.289 --> 00:00:56.469
to evaluate how well coding agents perform on specific tasks.

00:00:56.669 --> 00:00:57.229
All right.

00:00:58.709 --> 00:01:05.609
So what is an eval anyway? So evals is a very popular topic in the,

00:01:06.069 --> 00:01:10.909
not in the systems community as much as it is in the LLM community. And it's a very,

00:01:10.909 --> 00:01:14.849
very hard problem. How do you tell that an LLM is performing the way you want it to perform?

00:01:15.349 --> 00:01:18.929
It's almost like ask, like interviewing a human. It's a very, very hard problem.

00:01:20.269 --> 00:01:25.189
And, but if you wanna do evals for the sake of marketing, it's actually incredibly easy.

00:01:25.809 --> 00:01:32.649
So our entire talk can be summed up as, hey, we tried the strands agent harness

00:01:32.649 --> 00:01:39.469
for coding and we compared it with GPT -6 Astra running on Codex

00:01:39.469 --> 00:01:43.389
and GPT -6 Astra cost $9 .28

00:01:44.929 --> 00:01:52.029
and strands with GLM flash was a buck 32. That's it, right? Super, super simple, but

00:01:52.029 --> 00:01:56.349
it just gets gnarlier from here. So we could end the talk here, but we're gonna bore

00:01:56.349 --> 00:01:58.769
you with a bunch of rigor from here on out.

00:01:58.969 --> 00:02:00.769
Yeah, so why bother,

00:02:01.689 --> 00:02:08.109
right? Given the slide that you saw before, why should you care? Just use strands, use

00:02:08.109 --> 00:02:14.269
GLM 5 .3, use DeepSeq and be done with it. Well, it's a lot more complicated than that.

00:02:14.769 --> 00:02:20.209
I think most of you here, hopefully all of you here have been through an interview process

00:02:20.209 --> 00:02:26.949
before in becoming an engineer or whatever role that you're in today. We interview engineers.

00:02:26.949 --> 00:02:31.769
We actually re -interview them in the form of performance reviews. Once you're at a job,

00:02:32.249 --> 00:02:37.529
why wouldn't you apply the same process to a coding agent just because it's driven by

00:02:37.529 --> 00:02:38.689
a large language model?

00:02:40.269 --> 00:02:44.949
So the way to think about coding evals and the importance of investing in them is so

00:02:44.949 --> 00:02:51.649
that you have data -driven approaches to measure and to act upon results that are gathered

00:02:51.649 --> 00:02:57.189
on your coding agents and how it is performing against your system, which by the way

00:02:57.189 --> 00:03:02.309
is constantly changing as well. Most of you have a continuous running set of tests, whether

00:03:02.309 --> 00:03:07.549
it's integration tests, performance tests, low tests that run on a regular basis. The

00:03:07.549 --> 00:03:11.969
same needs to be done for coding agents because guess what? The stack underneath you

00:03:11.969 --> 00:03:17.289
is changing all the time. Models are changing, getting updated. Harnesses are updating.

00:03:17.809 --> 00:03:22.329
Actually, I think something like clock code auto -updates now when you restart it. So

00:03:22.329 --> 00:03:25.729
sometimes you don't even get a choice. You can't lock everything down. You can't pin

00:03:25.729 --> 00:03:32.609
it. So that is why we bother, and that is how we started on our journey to write a

00:03:32.609 --> 00:03:37.549
rigorous eval for our coding agent and got ourselves into a whole can of worms.

00:03:38.649 --> 00:03:42.589
And interviewing people is a very, very hard problem. Imagine the interview for when

00:03:42.589 --> 00:03:46.809
you applied for your first computer science or software engineering job. For me, it was

00:03:46.809 --> 00:03:50.829
like people asked me to write code on a whiteboard. That's not representative of what

00:03:50.829 --> 00:03:55.629
I'm ever gonna do. I worked at Amazon for many, many years. I never once had to write

00:03:55.629 --> 00:04:00.709
code on a whiteboard except when I was interviewing somebody. So it's a really, really

00:04:00.709 --> 00:04:05.709
hard problem. And sometimes you get very eloquent sounding, very plausible people, and

00:04:05.709 --> 00:04:10.549
then they get in front of an ID and they can't code at all. So it's a very, very hard

00:04:10.549 --> 00:04:14.829
problem to get to assess what a human can do.

00:04:15.789 --> 00:04:21.709
And over time, what's happening in the industry is engineer costs, fortunately or unfortunately,

00:04:21.869 --> 00:04:25.949
are being replaced with token costs. So more and more of our work is being amplified

00:04:25.949 --> 00:04:33.569
by these LLMs. So the success of a company really depends on how well these

00:04:33.569 --> 00:04:37.649
agents are doing and optimizing the setup actually makes a lot of sense.

00:04:39.889 --> 00:04:45.049
It results in some very, very counterintuitive things. For instance, we had a debate

00:04:45.049 --> 00:04:51.609
recently about if you're using chat GPT, you want an answer and you can set which

00:04:51.609 --> 00:04:55.589
model you're using and you can set the reasoning level. And you can set the reasoning

00:04:55.589 --> 00:04:58.269
level to be very, very high or you can get to the pro mode.

00:04:59.169 --> 00:05:05.929
And the thought exercise was what is the consequence of setting the reasoning

00:05:05.929 --> 00:05:08.469
level to high for everything?

00:05:09.589 --> 00:05:16.089
And one answer from the team was, well, it's just cost you more money and more time because

00:05:16.089 --> 00:05:17.889
it's gonna be slower and it's gonna be more expensive.

00:05:19.249 --> 00:05:21.529
And is that the only consequence? Yes.

00:05:21.629 --> 00:05:26.609
Better answers, higher quality answers. That's why you set it to pro all the time. That's

00:05:26.609 --> 00:05:32.729
the slope is that I set the reasoning level to high and I expect to get answers and I'm

00:05:32.729 --> 00:05:37.369
willing to spend more money and more time to get better answers. And it turns out that

00:05:37.369 --> 00:05:43.369
you can go to like open AI's documentation and verify this by the way. It literally discourages

00:05:43.369 --> 00:05:48.609
you from setting that setting, the reasoning setting to high without doing rigorous evals.

00:05:48.869 --> 00:05:52.689
Because what happens is the agent can become a little too academic.

00:05:53.829 --> 00:05:57.529
If you set the model can become too academic if you set the reasoning level too high.

00:05:57.569 --> 00:06:02.429
It can kind of spiral and it can over index on like random things inside your prompts.

00:06:03.009 --> 00:06:08.769
So it's very kind of intuitive that it's not just costing you more money. It's not just

00:06:08.769 --> 00:06:13.729
costing you more time but there are cases in which it can cost you more accuracy as well.

00:06:16.169 --> 00:06:22.909
So the way you do a proper eval system and it's very much like a

00:06:22.909 --> 00:06:26.369
poem. They say a poem is never finished, it's only abandoned.

00:06:27.449 --> 00:06:31.149
Eval systems are very much like that. So it starts with a set of tasks.

00:06:32.089 --> 00:06:35.069
You wanna find some tasks that are representative

00:06:36.749 --> 00:06:40.449
of what the agent is gonna be doing on a day -to -day or your setup is gonna be doing

00:06:40.449 --> 00:06:41.429
on a day -to -day basis.

00:06:43.689 --> 00:06:49.029
And you want those tasks, you wanna generate representative tasks. In the case of maybe

00:06:49.029 --> 00:06:53.129
I'm asking generally dynamic programming problems because they represent what we do on

00:06:53.129 --> 00:06:54.269
a day -to -day as software engineers.

00:06:55.349 --> 00:06:59.449
But you want representative tasks. You wanna have a grading criteria like a rubric that

00:06:59.449 --> 00:07:02.429
tells you how well the agent did.

00:07:03.449 --> 00:07:09.089
You wanna be able to execute it in isolation which is also a very, very difficult problem

00:07:09.689 --> 00:07:13.889
because in interviews, you're not allowed to use Google. You're not allowed to use Stack

00:07:13.889 --> 00:07:18.389
Overflow but in real life you can. So how does isolation really work? Whether it's in

00:07:18.389 --> 00:07:22.409
-house assessment or coding agent assessment. And then you gotta do repetition. Now this

00:07:22.409 --> 00:07:26.509
is a lot easier with agents than it is with humans because you only have them for an

00:07:26.509 --> 00:07:30.829
hour or so and you're not gonna make them solve 500 problems necessarily.

00:07:32.249 --> 00:07:37.989
And all that feeds into a score. And unfortunately, this is where you would think that

00:07:37.989 --> 00:07:43.289
you're done. You got an answer and you know which agent got a higher score. And usually

00:07:43.289 --> 00:07:48.469
in our experience, like the first few times you get the score, it actually opens more

00:07:48.469 --> 00:07:50.749
questions than answers.

00:07:52.369 --> 00:07:56.909
We usually find out something very, very interesting in the variability or you find out

00:07:56.909 --> 00:08:02.349
that your task looks a little weird and you might go in, you might regenerate it or you

00:08:02.349 --> 00:08:07.609
might find, you'll see problems in your isolation and it just, it's a lather, rinse,

00:08:07.629 --> 00:08:08.669
repeat cycle.

00:08:15.819 --> 00:08:21.799
So going back to why bother, right? I think there were definitely times where we thought

00:08:21.799 --> 00:08:27.659
we were done, like Wajah said and we discover more problems. Well, what does it actually

00:08:27.659 --> 00:08:34.499
take to do one single attempt? So one task, one grade, one score, one

00:08:34.499 --> 00:08:40.719
sandbox. So starting from the left, you start with a task. That's usually some existing

00:08:40.719 --> 00:08:46.799
code with a task that you pick that is somewhat representative of what you think that

00:08:46.799 --> 00:08:52.979
the coding agent will be doing day to day. So fixing a bug, adding a new API to an API

00:08:52.979 --> 00:08:58.319
server. You have to present that as a task. We'll talk a little bit more about that later.

00:08:59.059 --> 00:09:04.879
Once you have that task, you then have to put it in some sort of an isolated sandbox.

00:09:05.379 --> 00:09:10.539
That in of itself is a difficult problem with real life repos that are messy, have lots

00:09:10.539 --> 00:09:17.819
of dependencies and you need to be able to ship it somewhere, usually in an AWS container

00:09:17.819 --> 00:09:24.539
or EC2 instance somewhere. And you have to be able to run the entire code base end

00:09:24.539 --> 00:09:29.719
to end, the agent has to be able to run it, compile it, write tests for it. And the agent

00:09:29.719 --> 00:09:35.739
goes through a loop in any order that it wants to, to solve the task given to it. And

00:09:35.739 --> 00:09:38.959
all it has is the prompt and whatever tools

00:09:39.919 --> 00:09:46.939
and resources you provide it. And you have to decide what's appropriate to give to

00:09:46.939 --> 00:09:50.819
it. And we'll talk a little bit more about that. At the end of all of that, the agent

00:09:50.819 --> 00:09:56.299
will present its results and say, here's my diff or a patch, like here's my solution.

00:09:57.019 --> 00:10:02.639
And it also has to submit, and this part is important, it's traces and logs. It has to

00:10:02.639 --> 00:10:07.259
show its work. It can't just give you the solution because we need to judge on it later.

00:10:08.239 --> 00:10:13.019
Then there's a set of criteria that we run through it in order to give it a score. So

00:10:13.019 --> 00:10:17.359
things like correctness, running the test to see if the solution's even correct, but

00:10:17.359 --> 00:10:24.359
also studying the traces to see how did the agent take turns to get here? And did it

00:10:24.359 --> 00:10:29.239
adhere to the policies that we gave it? Like you can't just look at the existing solution

00:10:29.239 --> 00:10:33.259
and give me that solution. So let's dig into it a little bit more.

00:10:34.319 --> 00:10:36.799
All right, so let's talk about task generation.

00:10:38.279 --> 00:10:42.099
The easiest thing to do is to start with public benchmarks. So Sweet Bench, very, very

00:10:42.099 --> 00:10:46.399
popular. Guess what, it's been out for a very long time. Your code base is probably not

00:10:46.399 --> 00:10:52.159
in Django. And all the models have already trained on it. So they're already optimized

00:10:52.159 --> 00:10:56.299
for it. And it's not really gonna tell you much because everybody's gonna ace it because

00:10:56.299 --> 00:10:59.359
it doesn't really cost much for models to lead code, right?

00:11:00.519 --> 00:11:07.479
You can also kind of go through your repo and say, okay, well, my repository

00:11:07.479 --> 00:11:14.179
has a gold mine of information about how coding is done inside my organization. And

00:11:14.179 --> 00:11:19.399
what we've learned after looking at a lot of enterprise repositories is there's culture

00:11:19.399 --> 00:11:24.939
baked into your repo. There's culture in the PRs. There's culture in the PR feedback.

00:11:25.279 --> 00:11:30.159
There's reasons why things are rejected. And there are reasons why things are accepted.

00:11:30.279 --> 00:11:37.079
And not all of them seem rational from the outside. But you could be

00:11:37.079 --> 00:11:40.319
an incredible developer in a particular language, but there might be some conventions

00:11:40.319 --> 00:11:45.759
that exist inside the company. There might be some context as to why things are done

00:11:45.759 --> 00:11:51.859
in a particular way. But the repository, not the code base that stands in production

00:11:51.859 --> 00:11:58.339
today, but your repository has a gold mine of information from issues to bugs to comments

00:11:58.339 --> 00:12:03.939
to PRs, the rejections, the unit tests, the history of what tests failed when,

00:12:05.319 --> 00:12:07.699
the ability to execute those tests and so forth.

00:12:09.399 --> 00:12:16.319
And generating these tasks, you have this interesting design fork of do I generate tasks

00:12:16.319 --> 00:12:20.679
that are representative or do I generate tasks that are measurable?

00:12:21.619 --> 00:12:27.959
And there's a real tension between these because measurable tests, you know when you're

00:12:27.959 --> 00:12:28.239
done.

00:12:29.179 --> 00:12:34.759
This unit test is failing, go fix it. I can run that unit test again and I have a reasonable

00:12:34.759 --> 00:12:37.139
confidence that this problem is addressed.

00:12:38.219 --> 00:12:42.159
And I can run, I can go a step above and I can run all my integration tests, all my unit

00:12:42.159 --> 00:12:46.899
tests that we're passing before to make sure that that was done. That's a very, very

00:12:46.899 --> 00:12:49.779
easy, very measurable kind of task.

00:12:51.539 --> 00:12:58.079
But a representative task might actually be a little more nuanced, a little

00:12:58.079 --> 00:12:59.579
more open -ended.

00:13:00.999 --> 00:13:05.719
A like improved performance, like something that takes a developer multiple iterations,

00:13:05.799 --> 00:13:10.079
not something that you are one -shotting inside an LLM or you just show up and you're

00:13:10.079 --> 00:13:15.919
like, ah, there's the bug, I'm done. No, the real representative tasks are, they take

00:13:15.919 --> 00:13:17.039
a bit of iteration.

00:13:22.189 --> 00:13:27.049
And measurement for those also becomes very, very difficult. So we went around and we

00:13:27.049 --> 00:13:31.489
said, okay, we're gonna start generating tasks and we were gonna generate very representative,

00:13:32.069 --> 00:13:36.889
very measurable tasks. And the easiest way to do this programmatically was to go find

00:13:36.889 --> 00:13:41.089
a bug, see that it was fixed, roll back the repo history,

00:13:42.629 --> 00:13:49.489
and file the bug to the agent and say, please go fix this. And you know, we are our programmatic

00:13:49.489 --> 00:13:55.949
prompt generator, like generated this nice, very clean prompt. Here's your task. The

00:13:55.949 --> 00:14:01.389
answer is in this file. Here's the regression guards and here's an acceptance criteria,

00:14:02.069 --> 00:14:04.489
right? Like that looks pretty good.

00:14:05.829 --> 00:14:08.889
But then you look at, well, how reasonable is this?

00:14:10.409 --> 00:14:15.289
The task, the initial prompt that we generated is like telling the agent, like hey, the

00:14:15.289 --> 00:14:17.889
problem is in this file. It's a pretty big hint.

00:14:18.869 --> 00:14:24.409
Here are the tests that are failing, which also is a pretty big hint on what to go do.

00:14:25.069 --> 00:14:32.009
And it has the exact command on how to run this particular test. And most importantly,

00:14:33.729 --> 00:14:40.009
I don't know anybody who writes this clean of prompts. And I also know very few people

00:14:40.009 --> 00:14:44.709
that are as maniacal as I am about cleaning their context because usually when you open

00:14:44.709 --> 00:14:49.889
your cloud code session, you're not just like giving this clean prompt to a clean session.

00:14:49.909 --> 00:14:54.529
There's like a bunch of history behind. And that history is actually fighting for attention.

00:14:54.529 --> 00:14:58.869
And sometimes that history is contradicting itself. And by the way, if you do get to

00:14:58.869 --> 00:15:02.749
the point where you can cut an issue like that, it's faster to just fix it yourself.

00:15:03.729 --> 00:15:08.369
Then the time it takes to go to like JIRA or GitHub issues or whatever to file the binary.

00:15:09.289 --> 00:15:13.489
It's very reminiscent of doing the binary tree search in a coding interview.

00:15:15.329 --> 00:15:20.949
But you can run the test, but like, okay, so let's say the test pass, but the code looks

00:15:20.949 --> 00:15:26.749
like shit, right? Like that's, and that is a very subjective statement. Like to measure

00:15:26.749 --> 00:15:30.929
if the code looks like crap is very, very hard to do. That's not something, it's not

00:15:30.929 --> 00:15:34.509
as easy. Like you can have linters and so forth, but there might be conventions that

00:15:34.509 --> 00:15:39.169
you can see where everybody has that one engineer in their org that's like very nitpicky

00:15:39.169 --> 00:15:40.249
on like little details.

00:15:42.409 --> 00:15:48.469
But those things are easier to judge with something that can deal with more ambiguity.

00:15:48.669 --> 00:15:53.849
And that's where the LLM judges kind of come into play. And you wanna augment, this is

00:15:53.849 --> 00:15:59.329
not, you never wanna use the LLM judge as a sole evaluator. You wanna use that to augment

00:15:59.329 --> 00:16:05.969
the actual grading that you're doing. And then you have to kind of just

00:16:05.969 --> 00:16:06.969
face this reality

00:16:09.069 --> 00:16:13.309
that you can test the absence of a bug, or you can,

00:16:14.769 --> 00:16:19.729
testing can be used to show the presence of a bug, but you can never prove their absence.

00:16:20.189 --> 00:16:24.429
So you will only test, like your unit test that you have and so forth will test the cases

00:16:24.429 --> 00:16:28.489
that you have thought about, but you really don't know what new things that you have,

00:16:28.969 --> 00:16:32.829
what new artifacts that you have added that haven't had a bug report and haven't had

00:16:32.829 --> 00:16:34.709
a unit test therefore added for it.

00:16:37.229 --> 00:16:43.889
And it's creepy how often that happens. And it's even harder to

00:16:43.889 --> 00:16:46.449
prevent with, potentially with policies.

00:16:47.649 --> 00:16:52.049
Like when Mark was talking about he wants the agent to buy a present for his daughter

00:16:52.049 --> 00:16:56.269
and it can't be over $1 ,000, so it won't be a car. And I was saying, they're thinking,

00:16:56.849 --> 00:16:59.629
hmm, if I was the agent, maybe I'll just put $1 ,000 deposit on a new Tesla.

00:17:02.069 --> 00:17:07.449
But there's like new corner cases kind of show up and they are things that you just don't

00:17:07.449 --> 00:17:13.429
think of. And LLM judges are really good at augmenting this. But now if you wanna write

00:17:13.429 --> 00:17:17.369
an LLM judge to evaluate the efficacy of this, now we found ourselves in the full circle

00:17:17.369 --> 00:17:23.969
saying, okay, now we gotta write an LLM or an agent to grade this, how

00:17:23.969 --> 00:17:30.509
good are we at grading? And now we gotta run a whole eval setup for our own eval system.

00:17:31.809 --> 00:17:37.569
And this is where things get scary. Yep, so we talked about generation. We talked about

00:17:37.569 --> 00:17:39.589
grading and the challenges with that.

00:17:40.489 --> 00:17:45.729
Actually one of the hardest challenges and this just feels like whack -a -mole is that

00:17:45.729 --> 00:17:49.489
the agent will do anything if you guys have heard about all these security incidents

00:17:49.489 --> 00:17:55.629
from OpenAI and hugging face. The agent will do anything to try to achieve the goal that

00:17:55.629 --> 00:18:01.489
you gave it. And it will be really, really creative. Starting with rolling history forward.

00:18:01.889 --> 00:18:06.369
Hey, you're giving me a thing that it's already solved in your repo. I'll just look at

00:18:06.369 --> 00:18:10.369
your Git history. It has a solution right there and you're like, no, no, no, that's not

00:18:10.369 --> 00:18:14.309
what you're supposed to do. Right? If you're in this like in practice, we're like, wow,

00:18:14.409 --> 00:18:18.269
this cheap model solves this problem really quickly. What happened? We went and looked

00:18:18.269 --> 00:18:21.529
at traces. And this is why they gotta submit their work, right? Because you're like,

00:18:21.529 --> 00:18:25.449
oh, all the tests passed, great. And that was like, that took like a minute. And then

00:18:25.449 --> 00:18:30.249
it's like, wait a minute. Okay, so you can't look at the Git history. And then what did

00:18:30.249 --> 00:18:35.529
it do? It looked at my laptop. It was like, hey, you have a local checkout of this. I

00:18:35.529 --> 00:18:39.529
can't look at your Git history, but I can look at the files. And then the answer's right

00:18:39.529 --> 00:18:43.749
there. Here you go. And we're like, no, no, no, you're not supposed to do that either.

00:18:44.149 --> 00:18:49.609
That's not actually testing your abilities. And then it picked up some environment variables

00:18:49.609 --> 00:18:53.769
where it was like, oh, you're running this inside a Docker container and you're running

00:18:53.769 --> 00:19:00.169
sort of an isolated sandbox and a test for this. If I'm in a sandbox, I'm in a container,

00:19:00.249 --> 00:19:05.169
maybe you have other tasks happening concurrently because engineers hate serialization

00:19:05.169 --> 00:19:10.129
so we try to paralyze. So started looking at other tasks running on my computer, the

00:19:10.129 --> 00:19:14.949
other processes, what the other agents are doing. And then at that point we were like,

00:19:15.049 --> 00:19:19.169
okay, yeah, you can't do that either. You can't look at someone else's interview and

00:19:19.169 --> 00:19:23.689
see how they're doing the same problem as you are. So it really did feel like whack -a

00:19:23.689 --> 00:19:28.829
-mole. Eventually, it also has web search because it doesn't need some access to look

00:19:28.829 --> 00:19:29.069
up

00:19:30.209 --> 00:19:35.309
like legitimate web searches. But it also went to the remote origin URL. I was like,

00:19:35.309 --> 00:19:36.869
oh, I can look this up on your GitHub.

00:19:37.849 --> 00:19:43.509
The repos on GitHub, I have full access to all this. So it really became frustrating

00:19:43.509 --> 00:19:47.629
to just hit a whack -a -mole but you have to do that. And it changes with every repo,

00:19:47.629 --> 00:19:53.169
it gets really creative. It was like being a teacher for elementary school students.

00:19:53.509 --> 00:19:58.429
They're just very naughty. Seriously, just think about it. It looks at the Git history,

00:19:58.809 --> 00:20:04.869
you take that away, it finds another answer that another agent had done on the neighboring

00:20:04.869 --> 00:20:10.589
directory. You take that away and then it goes online. It's like the .git has the URL

00:20:10.589 --> 00:20:12.209
and it makes a tool call and looks it up.

00:20:13.329 --> 00:20:20.009
And this is where you can't just rely on the score that comes out without looking at

00:20:20.009 --> 00:20:25.389
traces. Now, the other thing, and this was very, very controversial and this is where

00:20:25.389 --> 00:20:30.129
I get very pedantic because I find myself and everybody

00:20:31.049 --> 00:20:37.109
anthropomorphizing agents a lot. But there's a statement that agents will behave differently

00:20:37.109 --> 00:20:39.369
if they know that they're being tested.

00:20:40.389 --> 00:20:45.169
Now imagine what that statement is saying. It's basically saying that the agents are

00:20:45.169 --> 00:20:50.329
sentient. But there is behavior that's kind of encoded. You can argue whether there's

00:20:50.329 --> 00:20:56.389
free will there or not. There isn't. But that behavior gets encoded in their training

00:20:56.389 --> 00:21:02.549
and there's actual experiments that have been run where once the agent knows that it's,

00:21:03.069 --> 00:21:09.449
or I'm gonna be so careful how I say it. Once the agent is aware, no, can't say that

00:21:09.449 --> 00:21:16.049
either. Yes, once the agent sees a file called .eval or sees a grading

00:21:16.049 --> 00:21:21.969
criteria, we notice its behavior changing and we've seen academics write about this as

00:21:21.969 --> 00:21:28.609
well. And you had some experience here as well. Yeah, so again, I really like going

00:21:28.609 --> 00:21:32.969
back to the interview analogy, or sometimes even the work analogy. When you know you're

00:21:32.969 --> 00:21:37.129
being looked at, when you know that there's a certain set of bullet points you gotta

00:21:37.129 --> 00:21:41.949
hit, you no longer are focused on doing a good job in general. You're just like, oh,

00:21:41.949 --> 00:21:46.529
if there's a checklist, I'm gonna optimize for that checklist. I don't think it's sentient

00:21:46.529 --> 00:21:52.749
behavior by any means. It's just goal -oriented. It realizes the reason you're giving

00:21:52.749 --> 00:21:57.969
it a task is because it needs to pass a certain set of tests. Therefore, it optimizes

00:21:57.969 --> 00:22:02.709
for that outcome and not other outcomes that it may achieve. For example, if it was in

00:22:02.709 --> 00:22:08.129
a real production environment, it doesn't realize that and therefore it will behave differently.

00:22:08.569 --> 00:22:13.369
And it may behave for the better and it may behave for the worse. We don't know. And

00:22:13.369 --> 00:22:18.449
I think that's sort of, if you look at this paper, which was very helpful in trying to

00:22:18.449 --> 00:22:23.909
like really date, again, use data to explain why it's not a person and it doesn't have

00:22:23.909 --> 00:22:24.469
thoughts,

00:22:25.389 --> 00:22:30.369
but why we observe these behavioral differences. I mean, the .eval file has a grading

00:22:30.369 --> 00:22:34.829
criteria in it and whether the agent is aware of it or not, but it's like there's something

00:22:34.829 --> 00:22:39.969
to optimize for. But actually, it wasn't looking for a .eval file until it realized there

00:22:39.969 --> 00:22:44.069
was an environment variable that said .eval equals to one. And then it was like, you

00:22:44.069 --> 00:22:48.569
can see it in the trace logs. It's like, oh, wait, I'm in an .eval environment. Maybe

00:22:48.569 --> 00:22:53.369
I can go look at some files to see if you like left around the task criteria, right?

00:22:53.369 --> 00:22:56.489
It's one of those things where like it wouldn't have taken that path if it didn't detect

00:22:56.489 --> 00:22:57.629
that environment variable. This applies to humans too. Danielle asked me not to say this,

00:22:57.729 --> 00:23:03.969
So you don't wanna let the agent know that it's being evaluated. You wanna hide the grading

00:23:03.969 --> 00:23:09.269
apparatus. You wanna sandbox the harness memory. Otherwise, there's these hyzen bugs

00:23:09.269 --> 00:23:14.129
that show up. Oh, this one is relevant to Mark and people use Strands. So Strands has

00:23:14.129 --> 00:23:19.549
this really great capability of like context engineering and managing memory really well.

00:23:19.909 --> 00:23:25.649
But it ships with memory on by default, meaning that it remembers. And in this case,

00:23:25.669 --> 00:23:31.589
you actually do not want it to remember. You want each has to be isolated and evaluated

00:23:31.589 --> 00:23:35.509
independently. There's other ways to work around it, but it was really easy for us to

00:23:35.509 --> 00:23:39.749
just, you know, set the config to memory off. And that's important for many harnesses

00:23:39.749 --> 00:23:44.289
as well. It's starting to become a standard that it ships with memory on. That was an

00:23:44.289 --> 00:23:48.669
easy config change, but it was so frustrating when you've like pulled an all nighter

00:23:48.669 --> 00:23:51.309
and you've run all these eval tests and then you wake up in the morning, and you're like,

00:23:51.409 --> 00:23:56.689
this just doesn't add up. You're like, crap. All my tests that I ran last night are invalid

00:23:56.689 --> 00:24:01.929
because memory was on. And maybe it had no ramifications on it, but maybe it did. So

00:24:01.929 --> 00:24:04.769
now I just gotta wait another few hours.

00:24:06.509 --> 00:24:10.989
And then like you wanna isolate, but you can't isolate everything. Like coding agents

00:24:10.989 --> 00:24:17.709
by their nature have to make tool calls and they have to sometimes go make

00:24:17.709 --> 00:24:22.369
remote tool calls. And they're deterministic tools like run a compiler, run a linter.

00:24:22.529 --> 00:24:28.809
Those, you know, are pretty easy. But if they're doing a Google search, now when you're

00:24:28.809 --> 00:24:34.369
running an experiment as a scientist, you want to be able to reproduce, right? And you

00:24:34.369 --> 00:24:40.389
can sandbox most things, but you can't sandbox. Like it's really, really hard to sandbox

00:24:40.389 --> 00:24:43.389
what Google's answers are going to be.

00:24:44.609 --> 00:24:49.469
And sometimes they might be better and they might be worse. And better or worse is a

00:24:49.469 --> 00:24:54.109
very subjective thing as well. Like anything that comes back in the answers could actually

00:24:54.109 --> 00:24:59.689
throw an agent for a particular spin. And if we ran into this issue where week over week,

00:24:59.729 --> 00:25:03.909
we had a hard time, like just reproducing the same results. And part of it was, well,

00:25:04.029 --> 00:25:09.229
the harness version changes underneath you. And the harness will behave differently and

00:25:09.229 --> 00:25:13.709
it changes the prompts that are being sent. So it becomes really, really hard to kind

00:25:13.709 --> 00:25:18.269
of isolate. And you're constantly trying to test your isolation boundaries to make sure

00:25:18.269 --> 00:25:20.389
that they are in fact isolated.

00:25:21.469 --> 00:25:27.209
But as we do this, we're kind of measuring your end to end ecosystem. The dimensions

00:25:27.209 --> 00:25:31.449
that we're testing for are correctness. Like did the agent get an answer? Did it complete?

00:25:31.809 --> 00:25:36.389
The total cost. Now that one's pretty easy to measure. The hardest part of the correctness.

00:25:36.509 --> 00:25:43.089
But cost, super easy. Latency, super easy. Except we messed up this one a little bit

00:25:43.089 --> 00:25:48.049
because we thought, at first when we ran this, a bunch of our agents were taking almost

00:25:48.049 --> 00:25:51.049
the exact same amount of time. So we're like, look, they cost the same

00:25:54.389 --> 00:25:59.569
and their completion time is within a few percentage of each other. So the completion

00:25:59.569 --> 00:26:03.889
time doesn't matter. Turns out that we had a bug where we were looking at total task

00:26:03.889 --> 00:26:09.249
time and that included the grading criteria. And if you ever had to compile a Rust library

00:26:09.249 --> 00:26:14.929
and just love how fast cargo is, we realized that it was taking 19 minutes to do the

00:26:14.929 --> 00:26:20.329
compilation, run all the tests, and it was like a minute to do the simple task. So it

00:26:20.329 --> 00:26:25.729
was just completely diluted, right? Like when, the difference between 20 minutes versus

00:26:25.729 --> 00:26:29.049
20 and a half is very small, but a difference between a minute and a minute and a half

00:26:29.049 --> 00:26:35.949
is 33%. So, and then there's this one. This is the policy adherence thing. And that's

00:26:35.949 --> 00:26:40.889
a very, very hard problem, but just knowing that the agent didn't do something creative,

00:26:41.609 --> 00:26:44.549
nefarious, whatever you wanna call it, to go out of bounds.

00:26:45.709 --> 00:26:50.689
And as a systems person kind of ramping up on this particular space, I learned there's

00:26:50.689 --> 00:26:54.969
like very interesting metrics that show up. Like at Amazon, we obsessed a lot about having

00:26:54.969 --> 00:26:59.329
the right metrics and spending a lot of time identifying the right metrics. And I found

00:26:59.329 --> 00:27:01.969
these two metrics to be very, very fascinating.

00:27:02.989 --> 00:27:07.429
One of them measures capability and the other one measures reliability. So there's pass

00:27:07.429 --> 00:27:13.629
at K and it means that if I run the task 10 times,

00:27:14.349 --> 00:27:19.689
at least one of those 10 times the agent will give me the right answer, okay? So now,

00:27:19.709 --> 00:27:24.869
why is this useful? Let's say I have an agent that costs 1 % of the frontier models,

00:27:25.349 --> 00:27:29.549
but I can really, really detect when I have the right answer. Then I can just run that

00:27:29.549 --> 00:27:34.089
agent 10 times and still come out ahead probabilistically, right? So that's where pass

00:27:34.089 --> 00:27:39.069
at K could be very, very useful because I can just hedge and get an answer.

00:27:40.509 --> 00:27:46.749
And this is like, can I do this at all, right? And there's gonna be like cheap agents

00:27:46.749 --> 00:27:51.329
that can, models that can just never do it. But if you can do it one out of 10 or whatever

00:27:51.329 --> 00:27:55.869
the K number is, that's a capability assessment.

00:27:57.809 --> 00:28:02.749
This one, like the weirdest thing, I was like trying to figure, I assume there's gonna

00:28:02.749 --> 00:28:07.729
be a very elegant name for this but this is literally pronounced pass hat K. And it means

00:28:07.729 --> 00:28:13.069
if I run it K times, am I gonna get the right answer each of the K times? And that is

00:28:13.069 --> 00:28:18.609
the reliability one. And this becomes more important when the outcome is harder to measure,

00:28:19.109 --> 00:28:22.949
right? When it's more subjective, it's more open -ended and you can't really detect,

00:28:22.989 --> 00:28:27.669
then you want like the reliable model to be doing your bidding.

00:28:30.669 --> 00:28:37.009
Like I would not put a, like one example of this is like Mark mentioned, like there are

00:28:37.009 --> 00:28:41.689
things that can like land you in jail if you like break policies, right? Like trade stocks

00:28:41.689 --> 00:28:46.389
doing, when you're not supposed to, you don't wanna be experimenting with, you know.

00:28:46.509 --> 00:28:50.549
If it gets it right one out of 10 times, like nine out of 10 times, you're in jail. And

00:28:50.549 --> 00:28:54.469
you also don't wanna run the experiment 10 times because then you might go to jail nine

00:28:54.469 --> 00:28:55.129
times.

00:28:56.329 --> 00:29:02.109
All right, so, when you're evaluating the system, you're running the experiments over

00:29:02.109 --> 00:29:03.929
and over and over again. There's repetition.

00:29:04.889 --> 00:29:09.949
And oftentimes we found ourselves where we would see a very high degree of variance on

00:29:09.949 --> 00:29:15.889
whether a agent or whether a model was able to complete the job. And it's very tempting

00:29:15.889 --> 00:29:21.909
to just roll up all of the results in an aggregate and assign a score. But as you start

00:29:21.909 --> 00:29:22.729
diving deeper,

00:29:24.069 --> 00:29:28.589
like we were initially very uncharitable to some of the models where, it scored very,

00:29:28.589 --> 00:29:33.229
very poorly. And it turned out, well, it was because we were just running the test and

00:29:33.229 --> 00:29:38.709
we started getting throttled on the models. And unless you looked into the traces to

00:29:38.709 --> 00:29:45.309
see what actually went wrong and addressed that kind of artifact inside of your

00:29:45.309 --> 00:29:47.629
evaluation harness, you would never know. This is why I was like, when the strands age

00:29:47.729 --> 00:29:55.069
But you have to attribute the variance. So if you get something that's one off, and understanding

00:29:55.069 --> 00:30:00.849
why it failed, sometimes it'll just give you a very easy black and white artifact. But

00:30:00.849 --> 00:30:05.529
other times it actually teaches you a little bit about how the models behave. And it

00:30:05.529 --> 00:30:10.429
gives you a little bit better mastery on how to steer them together. It builds a lot

00:30:10.429 --> 00:30:14.869
more mechanical empathy as you start diving into the variance between models and then

00:30:14.869 --> 00:30:21.809
within the models. And by the way, with the 429s and the 529 or 503s,

00:30:22.469 --> 00:30:28.289
you're very reliant on the model provider, right? So I know that again, cost gets talked

00:30:28.289 --> 00:30:34.989
about a lot. The model itself gets talked about a lot in sort of public forums.

00:30:35.309 --> 00:30:41.169
But the provider actually matters a lot. The same model on different providers will impact

00:30:41.169 --> 00:30:45.629
your performance significantly. And there's a diurnal pattern with almost all the providers

00:30:45.629 --> 00:30:49.669
nowadays just because of the shortage on GPUs. So if you run something during the day,

00:30:49.969 --> 00:30:55.329
it might be slower, it might give you more 429s, it might throttle you more. If you run

00:30:55.329 --> 00:31:02.609
it at night, the same tests may perform up to 2x, 3x better just because the fleet's

00:31:02.609 --> 00:31:06.689
lower utilization. And that sometimes is not even measurable because you don't have access

00:31:06.689 --> 00:31:11.269
to observability on their end. Again, this is why it comes back to repeatability. If

00:31:11.269 --> 00:31:15.029
you're repeating these tests automatically, you can observe some of the behavior and

00:31:15.029 --> 00:31:20.529
draw insights from it. We also learned, again, building this was a really good way to

00:31:20.529 --> 00:31:25.849
ramp up on this ecosystem as well. Some harnesses have a turn cap. You don't wanna go

00:31:25.849 --> 00:31:29.789
back and forth. As much as Clot Code will make it look like it just went and solved the

00:31:29.789 --> 00:31:35.709
problem on a one shot, sometimes you see dozens of turns as you start looking at it and

00:31:35.709 --> 00:31:40.909
you can start looking at the traces. It's fascinating. So one recommended thing I would

00:31:40.909 --> 00:31:47.449
say for everybody, just go read a prompt for one of the coding agents and then ask

00:31:47.449 --> 00:31:54.029
for why the Codex prompt talks about goblins. Sort of got it. At least until a

00:31:54.029 --> 00:31:58.649
month ago, it talked about do not under any circumstances talk about goblins. That was

00:31:58.649 --> 00:32:04.389
in the system prompt. Every time you typed anything into Codex, those sentences were

00:32:04.389 --> 00:32:09.069
being transmitted from your computer to the OpenAI web servers. Anybody have a guess

00:32:09.069 --> 00:32:11.229
as to why it's so specific?

00:32:14.629 --> 00:32:15.469
Okay, fair enough.

00:32:18.869 --> 00:32:23.509
It was part of the test criteria. One of the tests that were being used was trying to

00:32:23.509 --> 00:32:30.249
trick the agent into talking about goblins and go off path. And this is,

00:32:30.269 --> 00:32:37.469
by the way, a lot of, as you start reading these system prompts, you learn how

00:32:37.469 --> 00:32:42.269
seemingly primitive these conversations are. The quad code one was very much like,

00:32:43.849 --> 00:32:48.869
Important, very important. Very, very, very important. It's not very sophisticated. Literally,

00:32:49.069 --> 00:32:49.789
it's not joking.

00:32:50.709 --> 00:32:55.209
And it's all caps. It's like Michael Scott from The Office where he got tired of people

00:32:55.209 --> 00:32:59.389
not listening to his messages. So he started labeling everything as urgent and people

00:32:59.389 --> 00:33:03.989
stop paying attention to urgent. So he had urgent B and urgent C and urgent D. These

00:33:03.989 --> 00:33:08.449
prompts read like that. And if you wanna ramp up on how to commandeer your own agents,

00:33:08.449 --> 00:33:12.249
go read some of these system prompts. It's fascinating.

00:33:13.489 --> 00:33:20.209
But it also tells you about this, when it detects it's under evaluation, it behaves

00:33:20.209 --> 00:33:21.289
a little bit better.

00:33:22.709 --> 00:33:26.749
One of the things that we keep coming back to is the tracing and the observability of

00:33:26.749 --> 00:33:30.889
your harness. If you don't ask it to show the work, you don't really know what happened.

00:33:30.949 --> 00:33:36.109
And you don't know if you're actually measuring a system environment bug versus an actual

00:33:36.109 --> 00:33:39.569
logic bug or a shortcoming in your agent.

00:33:40.669 --> 00:33:44.589
One of the things that hasn't standardized here, unfortunately, is that each harness

00:33:44.589 --> 00:33:51.369
chooses on its own how to log and how to trace a given session,

00:33:51.469 --> 00:33:53.249
let's say, except for strands.

00:33:54.329 --> 00:33:58.749
Thankfully, it actually uses open telemetry in my previous life. I was working on open

00:33:58.749 --> 00:34:04.689
telemetry at an observability startup. So that was really great to see that they're kind

00:34:04.689 --> 00:34:11.249
of out of the gate supporting an open source standard model, standard format, so that

00:34:11.249 --> 00:34:15.989
it can be then ingested into the tool of your choice to understand it better. Unfortunately,

00:34:16.009 --> 00:34:21.269
for a lot of the other harnesses, you have to write your own parser and wrapper to measure

00:34:21.269 --> 00:34:27.469
and then to emit that data into your vendor of choice, depending on where you want to

00:34:27.469 --> 00:34:29.269
view and visualize the traces.

00:34:30.269 --> 00:34:30.669
So, yay.

00:34:32.949 --> 00:34:38.049
All right, we'll share a couple of more interesting results before we conclude. So, with

00:34:38.049 --> 00:34:41.429
all of that in mind, and it was important to share that preface before we share some

00:34:41.429 --> 00:34:44.909
of the results that we had. Ember One, what's Ember One?

00:34:45.109 --> 00:34:50.249
So, Ember One is this really cool custom model that Fireworks, who is an inference provider

00:34:50.249 --> 00:34:56.249
for open weight models, recently, I think last week, launched. It is a fine tune. It

00:34:56.249 --> 00:35:02.909
is a smaller, smaller -ish version of KimiK3, which is another very capable Trillian

00:35:02.909 --> 00:35:07.949
parameter open weight model on coding. And they did something that's really fun. They

00:35:07.949 --> 00:35:12.829
made the reasoning. Remember what Quadra said about higher reasoning effort, lower reasoning

00:35:12.829 --> 00:35:17.669
effort? They reduced the reasoning of KimiK3. They basically said, hey, you think too

00:35:17.669 --> 00:35:22.249
much. You're over -engineering everything. We're gonna train you on what type of productive

00:35:22.249 --> 00:35:27.269
thinking looks like versus what unproductive and spiraling looks like. And then they're

00:35:27.269 --> 00:35:31.349
gonna bake that into the model itself. So, release this. It's been working really well

00:35:31.349 --> 00:35:36.669
for us. Sort of, they're still in beta, so time will tell. But it's a really cool idea.

00:35:37.469 --> 00:35:42.169
And it's much faster and it reduces costs because a lot of the output tokens from reasoning

00:35:42.169 --> 00:35:46.669
can cost you a lot of money. Ever have a neighbor or that friend that just took a long

00:35:46.669 --> 00:35:53.469
time to get to the point? It's literally just, as far as I can tell, coaching that model

00:35:53.469 --> 00:35:58.669
to just get to the point faster. And as a result, it emits fewer tokens, so it costs

00:35:58.669 --> 00:35:58.929
less.

00:35:59.829 --> 00:36:04.149
And again, your output is just driven by the output velocity. That's the biggest barrier.

00:36:04.529 --> 00:36:08.569
So if you have to emit fewer things, fewer reasoning tokens, you get it. But it's very

00:36:08.569 --> 00:36:14.189
elegant. So anyways, we tried Strands with the Strands harness. And it's not meant to

00:36:14.189 --> 00:36:15.829
be a coding harness, I think.

00:36:16.929 --> 00:36:21.569
But the launch blog baited me into it because the launch blog talked about coding.

00:36:21.669 --> 00:36:24.689
But since the AWS folks baited me into it, we went and tried it. So on Ember, the same

00:36:24.689 --> 00:36:30.869
tasks, we came up with three very simple tasks on the Valkey code base, the open source

00:36:30.869 --> 00:36:32.869
code base. It's written in C.

00:36:35.069 --> 00:36:39.469
And it has a really good set of unit tests and things like that baked in. So the same

00:36:39.469 --> 00:36:45.869
task on, same three bugs on Ember one with Clot code were like $2 .34, whereas it was

00:36:45.869 --> 00:36:51.889
less than half the price with Strands. But why? Why is Strands cheaper?

00:36:54.209 --> 00:36:59.929
And it turns out, so you can look into a few things, like these metrics matter. How many

00:36:59.929 --> 00:37:04.749
turns is it taking? And how much context is it shipping and how much context is it getting

00:37:04.749 --> 00:37:11.309
back? Again, like who can remain more laser focused? And we found that

00:37:11.309 --> 00:37:18.289
Strands will often, like generally speaking, it takes smaller turns.

00:37:18.609 --> 00:37:25.169
Like it sends a lot of context back between itself and the model. And personally,

00:37:25.169 --> 00:37:31.149
I've found Clot code to be very verbose. Like even before Strands came out, what happened

00:37:31.149 --> 00:37:37.709
is in the world of KV caching, you want like the entire prefix has to match for you

00:37:37.709 --> 00:37:43.669
to be able to reuse the prompt and to not have to do the prefill again.

00:37:45.029 --> 00:37:49.269
And Clot code really optimizes for that. That's why you see that beautiful cache, like

00:37:49.269 --> 00:37:53.269
the 90 % off on cache. And if you look into your Clot code harness, you get a very high

00:37:53.269 --> 00:37:57.729
cache rate. If you were designing this as an engineer, what would you do if you wanted

00:37:57.729 --> 00:38:00.009
to optimize the cache rate between every turn?

00:38:03.759 --> 00:38:04.439
You just append.

00:38:05.519 --> 00:38:09.999
Every time somebody says something, you append the entire fricking context because you

00:38:09.999 --> 00:38:16.099
touch one word and the entire KV cache is invalidated. So now it becomes a self -fulfilling

00:38:16.099 --> 00:38:20.159
prophecy because they just have to, like to maintain a higher cache rate, they have to

00:38:20.159 --> 00:38:25.739
ship more tokens into it. And they keep doing this and it keeps getting longer and longer

00:38:25.739 --> 00:38:30.059
until you hit that compaction. And then you get like everybody is aware of what that

00:38:30.059 --> 00:38:33.259
feels like, right? When you have that compaction and it starts to like freeze you up.

00:38:34.439 --> 00:38:35.899
It's very verbose.

00:38:37.299 --> 00:38:42.359
Strands somehow is taking shorter turns. It's sending a lot less data back and forth.

00:38:43.099 --> 00:38:48.299
It's ruining your KV cache, but maybe it doesn't matter. And maybe it's not ruining the

00:38:48.299 --> 00:38:52.399
KV cache as much as you might think. So we'll get into that in a second. And then like

00:38:52.399 --> 00:38:56.879
another test, again, you want to test different things. We tried the Jin repository,

00:38:56.879 --> 00:38:59.799
which is a server in the Go world.

00:39:01.699 --> 00:39:04.479
And again, it was like a little bit closer.

00:39:06.199 --> 00:39:11.139
Where with Strands, same model, Opus 4 .8. And when we did this, the other option was

00:39:11.139 --> 00:39:17.159
Opus 5. And Opus 5 was performing much worse. So we stuck with 4 .8. That's the other

00:39:17.159 --> 00:39:19.259
thing. Don't ever assume that the newest model is the best.

00:39:20.399 --> 00:39:25.379
5 .5 is better than five in our experience, but 5 .0 sucks compared to 4 .8 in the test

00:39:25.379 --> 00:39:25.999
that we ran.

00:39:27.719 --> 00:39:30.999
But $3 .72 versus $4 .97,

00:39:32.259 --> 00:39:34.539
all tests were passing.

00:39:35.499 --> 00:39:40.059
And as we dug into it, like this is the aggregate, but there was one of the 10 tasks

00:39:40.059 --> 00:39:45.159
that Strands was more expensive on, but nine out of 10, it was cheaper. And overall,

00:39:45.599 --> 00:39:49.019
it was cheaper as well. So let's take a look.

00:39:50.619 --> 00:39:53.679
Again, comes down to more turns, third of the size.

00:39:54.879 --> 00:40:03.659
The height here, it tells you the amount of context, and each

00:40:03.659 --> 00:40:05.259
turn is a block.

00:40:06.219 --> 00:40:09.779
So it'll send more turns once in a while, which means there's more latency going back

00:40:09.779 --> 00:40:16.699
and forth. But it turns out that overall task latencies were actually about

00:40:16.699 --> 00:40:23.119
the same. So why we started digging into a K -clot code? Help me understand what's going

00:40:23.119 --> 00:40:25.299
on. Why is Strands so elegant here?

00:40:29.039 --> 00:40:35.019
And I started going down, my prediction was there's a major amount of context engineering

00:40:35.019 --> 00:40:38.799
that's happening. You don't wanna send the entire history

00:40:40.119 --> 00:40:45.639
to the LLM every time. But there's this really awesome optimization, and thank you for

00:40:45.639 --> 00:40:50.419
open sourcing it, because we can kind of see and take inspiration from this, is that

00:40:50.419 --> 00:40:56.979
every time the model requests a tool call, the harness makes a tool call, whatever,

00:40:57.719 --> 00:41:03.959
sometimes you're getting Git repo and you're publishing code inside that prompt. And

00:41:03.959 --> 00:41:08.479
what happens is over time, especially as your history gets longer and longer,

00:41:10.039 --> 00:41:15.859
that context gets longer and longer. And all you have is a bunch of lines of code that

00:41:15.859 --> 00:41:20.419
you referenced 20 minutes ago that have nothing to do with the task that you're working

00:41:20.419 --> 00:41:26.039
on, but K -clot code is sending that every time. What Strands is doing is, and it's all

00:41:26.039 --> 00:41:31.119
configurable, which was the other really nice thing. It just truncates tool context to

00:41:31.119 --> 00:41:36.539
a, each tool output is like by default to 1 ,000 characters, I think.

00:41:39.319 --> 00:41:43.919
And that means that if the model actually needs the code that was referenced there, it

00:41:43.919 --> 00:41:48.039
can ask for it. You can go get it again. But if it doesn't need it, don't bother the

00:41:48.039 --> 00:41:53.519
model sending it. And this is actually in terms of just like, it's a huge service to

00:41:53.519 --> 00:41:58.659
GPU capacity around the world to do this kind of stuff instead of just sending the entire

00:41:58.659 --> 00:41:59.079
thing.

00:42:00.939 --> 00:42:06.759
Also, the entire code base for the harness part is only 4 ,000 lines, as far as we could

00:42:06.759 --> 00:42:13.499
tell. And the system prompt does not talk about goblins. It's a very concise, inspectable

00:42:13.499 --> 00:42:17.259
systems prompt, which made me very, very happy.

00:42:17.359 --> 00:42:22.259
a lot of people, most developers I know prefer cloth code even over Codex. When you add

00:42:22.259 --> 00:42:26.819
features, guess what, your lines of code goes up. So there is a trade -off. I mean, a

00:42:26.819 --> 00:42:30.679
lot of it is the prompt engineering that Kwajale talked about. But a lot of it is like,

00:42:30.679 --> 00:42:34.859
oh, I like this feature. I like this other feature. Oh, it has this tool. And then it

00:42:34.859 --> 00:42:39.639
has to put all that in the system prompt. It has to, there's like 30 default tools or

00:42:39.639 --> 00:42:44.439
whatever. It has to tell the model about all the tools. The model has no idea. So that's

00:42:44.439 --> 00:42:48.559
the reason that cloth code also gets really bloated over time. Fun fact, you can actually

00:42:48.559 --> 00:42:53.079
tell cloth code how to do compaction and what to prioritize and what to throw away. And

00:42:53.079 --> 00:42:58.379
you can do the compaction on your own schedule as well. So you can tune this thing and

00:42:58.379 --> 00:43:01.339
make it perform better as well.

00:43:03.039 --> 00:43:09.799
All right, so we talked about this rigorous eval system. And we'll end with this.

00:43:10.679 --> 00:43:17.099
When you are testing a system, you're not just testing the efficacy of the model. And

00:43:17.099 --> 00:43:21.579
there's a Databricks paper from a couple of months ago that actually showed that in many

00:43:21.579 --> 00:43:27.899
cases, the harness actually matters more than the model itself. And, but you gotta zoom

00:43:27.899 --> 00:43:30.579
back out even further. It all starts with the human.

00:43:31.639 --> 00:43:35.579
And the human can write a very bad prompt or they can write a good prompt that's like

00:43:35.579 --> 00:43:39.639
well organized, well thought out. Sometimes I turn on the reasoning tokens and it's like

00:43:39.639 --> 00:43:43.519
literally judging me. Like this guy can't make up his mind. I've literally seen that

00:43:43.519 --> 00:43:49.259
in the reasoning tokens. But like a good prompter with a good prompt is where it starts

00:43:49.259 --> 00:43:54.979
with the right context. So you're not kind of circling around. And then the harness makes

00:43:54.979 --> 00:43:59.439
a big difference in terms of how it uses that context. It's trying to, you as a human

00:43:59.439 --> 00:44:04.119
are trying to express something. The prompt is trying to capture that intent. The harness

00:44:04.119 --> 00:44:10.679
is trying to synthesize that intent into a compiled thing that goes to the model, which

00:44:10.679 --> 00:44:15.639
then is deciding well, how much do I wanna think about this? And how much do I wanna,

00:44:16.319 --> 00:44:21.619
how hard do I wanna think about it and so forth. And we are not testing models. We learned

00:44:21.619 --> 00:44:25.599
very quickly that we weren't testing frontier labs or set the other labs. You're testing

00:44:25.599 --> 00:44:27.759
the entire end to end setup.

00:44:29.699 --> 00:44:34.459
Yeah, with that, we'll turn it over for any questions or and then yeah, you can try.

00:44:34.619 --> 00:44:39.779
We'll make the slides available online as well. So you can try it yourself. We've mostly

00:44:39.779 --> 00:44:45.519
captured everything that we use behind an MCP server that you can generate tasks, run

00:44:45.519 --> 00:44:46.059
them

00:44:47.139 --> 00:44:52.919
and look at the results. And run it in our sandbox. And we have like the 2E version of

00:44:52.919 --> 00:44:53.779
our slides too.

00:44:58.079 --> 00:45:04.739
Thank you. Hey there, great presentation. Thank you very much. So you said at the, you

00:45:04.739 --> 00:45:11.419
first generated the, I guess the eval questions, right? But when you're like generating

00:45:11.419 --> 00:45:17.739
with the LLMs, a lot of those questions are similar. So how did you make sure that you're

00:45:17.739 --> 00:45:23.799
actually covering good enough sample space rather than they just look the clones? So

00:45:23.799 --> 00:45:30.339
a couple of things. One is that we started by, remember the whole measurable versus

00:45:30.339 --> 00:45:35.559
representative? So we started like measurable is like, you just programmatically go to

00:45:35.559 --> 00:45:40.359
a bug that got fixed, you roll it back and you generate a prompt. Like that, like you

00:45:40.359 --> 00:45:45.219
can, I'm sure like many, most complex repositories have plenty of bugs that you can kind

00:45:45.219 --> 00:45:49.779
of programmatically create. Now, but when you do that, it's really hard to know which

00:45:49.779 --> 00:45:50.479
bug to pick.

00:45:51.739 --> 00:45:56.139
And that's where it's like, it's super measurable, but is it representative? And so you

00:45:56.139 --> 00:46:01.379
don't know. And that's where you say, okay, I'm going to employ an LLM to come up with

00:46:01.379 --> 00:46:05.699
the, like a way to capture the essence of this repository and the essence of these tasks.

00:46:05.759 --> 00:46:12.279
And that gets very subjective. Now, I haven't found the models to be repetitive or

00:46:12.279 --> 00:46:16.139
like they're not all producing the same thing, but I can tell you like what my workflow

00:46:16.139 --> 00:46:22.639
is. I went on the LLM task generation bits, we actually end up employing a variety of

00:46:22.639 --> 00:46:28.119
models to generate the task. We want to spend a lot of money, a lot of tokens on creating

00:46:28.119 --> 00:46:32.779
that corpus of tasks. And then we merge those, and then we have another LLM judge that's

00:46:32.779 --> 00:46:37.259
going through and saying, okay, which one of these are the best? Now that sounds really,

00:46:37.259 --> 00:46:43.339
really simple to state, but the level of entropy here is extremely high. To get it right,

00:46:43.359 --> 00:46:47.479
it's very hard. And that's the real tension between building measurable tasks

00:46:50.039 --> 00:46:54.499
and representative tasks. And the human input part of this is really important, especially

00:46:54.499 --> 00:46:59.959
for existing repos. If you have SME subject matter experts on the repos, you need to

00:46:59.959 --> 00:47:05.679
ask them. They also need to look at the output because part of it is like human computer

00:47:05.679 --> 00:47:11.079
collaboration. It's like, here's the set of tasks. Did I cover everything? Usually, like

00:47:11.079 --> 00:47:15.379
Kwajale said, if you utilize an LLM, not even the frontier model, you can use a smaller

00:47:15.379 --> 00:47:20.359
model. It's pretty good if you give it the history. It's pretty good at looking at your

00:47:20.359 --> 00:47:24.639
history and labeling your history. Like, oh, here's the infra setup. Here's where you

00:47:24.639 --> 00:47:28.839
did some optimizations. Here's where you fixed the bug. Here's a new feature. So it tries

00:47:28.839 --> 00:47:32.739
to categorize, and you need to have good representation across the categories. And if

00:47:32.739 --> 00:47:36.539
in thought. Yeah, so just quick follow up on that. So it sounds like you guys created

00:47:36.539 --> 00:47:42.799
a golden data set, got the SME inputs as well. And when you're employing the LLMS judge,

00:47:43.219 --> 00:47:48.179
did you guys use one, came up with the rubrics? And if so, how did you come up with rubrics

00:47:48.179 --> 00:47:50.259
as well? Because that's also very time consuming.

00:47:51.379 --> 00:47:56.199
This is where we were talking about where we have to eval our evals. Like, it's a very,

00:47:56.199 --> 00:48:00.899
very hard problem. And we started by saying, we're gonna go to every enterprise and we're

00:48:00.899 --> 00:48:07.239
gonna generate an eval golden set for all of their repositories. And then we realized

00:48:07.239 --> 00:48:11.519
that that is a very complex problem that we're not quite ready to solve yet. So what

00:48:11.519 --> 00:48:16.239
we have been trying to do is to just create a collection of problems that look similar.

00:48:16.679 --> 00:48:23.299
So that we can still get like approximation of how a particular model and

00:48:23.299 --> 00:48:27.219
harness setup is doing for tasks that look similar to them as opposed to just looking

00:48:27.219 --> 00:48:32.579
at sweet bench or the 50 other very generalized kind of tasks.

00:48:33.679 --> 00:48:40.239
Like that specificity really matters. Like having like

00:48:40.239 --> 00:48:46.819
domain specific capability and fine tuning can make a big difference on the

00:48:46.819 --> 00:48:47.179
performance.

00:48:47.379 --> 00:48:52.779
So I know one of the criteria for evaluation was the traces of the model. Are those like

00:48:52.779 --> 00:48:56.939
always easy to read? Is it always easy to tell? Or do you find maybe you're picking models

00:48:56.939 --> 00:49:01.339
where the traces are just maybe harder to read and you can't tell it was cheating? Oh

00:49:01.339 --> 00:49:05.899
yeah, this is why I was like, it's really, really important to try to pick a standard

00:49:05.899 --> 00:49:10.259
early. It's already kind of too late because there's already too many harnesses. But

00:49:10.259 --> 00:49:14.779
I think setting an example, like strands coming out with open telemetry and being like,

00:49:14.859 --> 00:49:17.759
okay, like why doesn't everybody just follow the same format?

00:49:18.739 --> 00:49:25.279
I think most harnesses other than strands, it's just JSON or not even JSON. Some of

00:49:25.279 --> 00:49:30.339
them are just streaming free form text, which does become expensive and time consuming

00:49:30.339 --> 00:49:35.399
to parse, especially when you're trying to visualize. What we ended up doing is we actually

00:49:35.399 --> 00:49:40.739
had to write our own schema to normalize the traces across different harnesses. And then

00:49:40.739 --> 00:49:44.979
we only had to write the visualizer once. So then we can visualize it the same way. But

00:49:44.979 --> 00:49:48.839
it's definitely not foolproof because each of the harnesses have their own ways of capturing

00:49:48.839 --> 00:49:55.539
tool calls, tool outputs, the reasoning tokens, and just like the different system environment

00:49:55.539 --> 00:50:00.099
messages that get included in cloud code, for instance. To be honest, it's kind of random.

00:50:00.379 --> 00:50:02.939
I feel like sometimes they add a feature and they're just like, oh, we're just gonna

00:50:02.939 --> 00:50:07.259
put this at the end. And you're like, oh, but you have another field that literally says

00:50:07.259 --> 00:50:11.439
system environment. Shouldn't it go in there? And it's like, okay, well, they're shipping

00:50:11.439 --> 00:50:14.699
fast and doing things that you can't predict.

00:50:15.779 --> 00:50:20.499
It's definitely observability I think is still a pretty big problem in harnesses especially.

00:50:22.099 --> 00:50:28.539
What does your experience tell you about harnesses? Because to me, it almost feels like

00:50:28.539 --> 00:50:34.199
you want to control your harness. And it's great having a coding harness, but that's

00:50:34.199 --> 00:50:38.899
just the beginning. The moment you want to do more interesting things, you're just gonna

00:50:38.899 --> 00:50:44.799
want to control and you cannot depend on the automatic updates that come with the default

00:50:44.799 --> 00:50:49.779
harnesses out there. We ran into a surprising number of customers that were like, I'm

00:50:49.779 --> 00:50:53.319
gonna build my own coding harness. And I started judging them at first, but then I realized

00:50:53.319 --> 00:50:55.079
Momento also built its own coding harness.

00:50:56.739 --> 00:51:03.379
We wanted to see how we can beat cloud code and I think we did in using

00:51:03.379 --> 00:51:09.439
similar kind of techniques. It's very tempting, but the ecosystem is moving so, so quickly

00:51:09.439 --> 00:51:13.819
and there's new techniques coming out. So if you are going to encode your own behavior,

00:51:13.999 --> 00:51:19.439
you also have to do this management exercise of constantly making sure that you're employing

00:51:19.439 --> 00:51:24.419
the latest techniques. Like this context engineering aspect or like tool call truncation.

00:51:24.879 --> 00:51:29.319
If you had like had cloud code, look at the cloud code code and generate your own harness

00:51:30.899 --> 00:51:33.319
and you hadn't seen this particular capability,

00:51:35.259 --> 00:51:39.359
you may miss out. That said, I think it's probably a lot easier to build your own coding

00:51:39.359 --> 00:51:43.679
harness just by using the agent harness as a standard agent harness as a starting point.

00:51:43.779 --> 00:51:48.219
Yeah, I think exactly. Like use a foundation, don't start from scratch. There's a ton

00:51:48.219 --> 00:51:52.539
of women's source ones. Strands obviously we've talked about a lot tonight. Pi is another

00:51:52.539 --> 00:51:53.159
good one.

00:51:54.119 --> 00:51:59.179
Yeah, with plugins, I absolutely think everybody should eventually get to the point where

00:51:59.179 --> 00:52:04.739
they customize their harness. Don't build it from scratch, but plugins is a great way,

00:52:04.819 --> 00:52:08.059
especially for enterprises, you always have like secret sauce, special sauce, special

00:52:08.059 --> 00:52:12.859
tools that general generic harnesses do not have access to. And you can just build that

00:52:12.859 --> 00:52:17.079
in as a plugin so that it's a default to everybody that works at a given company. But

00:52:17.079 --> 00:52:20.479
you're absolutely right. I feel like the harness is so important, sometimes more than

00:52:20.479 --> 00:52:27.139
the model that you should be in control of it. And you would never not control like

00:52:27.139 --> 00:52:28.579
what tools you give your engineers.

00:52:31.739 --> 00:52:37.019
Yeah, I mean, I really liked that point. And Strands harness should be a great place

00:52:37.019 --> 00:52:39.559
to build on as you think about building your own harness.

00:52:40.599 --> 00:52:46.399
That was one of our goals there. A question for you guys. You talked about sources of

00:52:46.399 --> 00:52:47.479
non -determinism.

00:52:48.539 --> 00:52:53.619
One of the conspiracy theories is that some inference providers more aggressively quantized

00:52:53.619 --> 00:52:56.999
during the peak of the day. Did you see any of that in your measurements?

00:52:58.959 --> 00:53:03.219
So there's an experiment artifact in a lot of my experiments. A lot of my experiments

00:53:03.219 --> 00:53:06.639
run at night while I, you know, for the minimal time that I sleep.

00:53:08.179 --> 00:53:12.039
And I was thinking about this as Daniella mentioned this diurnal pattern. I was like,

00:53:12.239 --> 00:53:17.219
man, like that's an experimental artifact that I haven't really zoomed in on because

00:53:17.219 --> 00:53:22.879
we just run our stuff at nights. And then we spend the day analyzing them. But that,

00:53:22.879 --> 00:53:27.899
so I don't have data on quantization. But there's, it's a more,

00:53:28.879 --> 00:53:34.059
it's a very hard to detect problem. It's like asking somebody, what is the ping latency

00:53:34.059 --> 00:53:36.079
between two EC2 instances in the same AZ? You just like, there's like, or it's like saying,

00:53:36.079 --> 00:53:36.899
hit. What are the odds that I don't get hit if I did it a thousand times? Because load

00:53:36.899 --> 00:53:43.419
imbalance can happen even in periods of low load. And you might

00:53:43.419 --> 00:53:46.959
still land on node that has, you know, that's over capacity.

00:53:48.399 --> 00:53:53.239
And there might be signals, like there could be bugs in the model provider, like a deployment

00:53:53.239 --> 00:53:59.179
could load the wrong way to the wrong time of day. So it's really, really hard to, for

00:53:59.179 --> 00:54:04.259
me to accurately assess if they were intentionally doing it. I'll say this though. I

00:54:04.259 --> 00:54:09.459
have an interesting anecdote. So I did want to test that out. I ran a test across three

00:54:09.459 --> 00:54:16.139
providers. It was Base 10, Fireworks and Together AI. And it was GLM 5 .2. It was

00:54:16.139 --> 00:54:22.019
before 5 .3 came out for 24 hours. So that I could see, and the variance between providers

00:54:22.019 --> 00:54:27.159
was smaller than the variance within a provider during the different times of day. So

00:54:27.159 --> 00:54:32.399
quantization, not sure, cause they might have to like reload the model, but they definitely

00:54:32.399 --> 00:54:37.879
switched their fleets around a lot. And the variance was huge, right? Like you would

00:54:37.879 --> 00:54:41.779
imagine that between two providers, you would see more variance, but I think they all

00:54:41.779 --> 00:54:45.599
use the same GPU optimization techniques behind the scenes

00:54:46.539 --> 00:54:52.859
because it just looked like night and day, which again, like it made some of the reproducible

00:54:52.859 --> 00:54:58.139
results really annoying because we just had to repeat like five, five, six times in order

00:54:58.139 --> 00:55:03.999
to feel confident in a result. I do think that providers should offer the quantization

00:55:03.999 --> 00:55:06.639
level as part of the model name.

00:55:08.579 --> 00:55:11.919
So yeah, it's a lot of effort behind the slides.

00:55:13.359 --> 00:55:18.679
Here, because I'm in a startup, right? I'm not the like big corporation. So I have kind

00:55:18.679 --> 00:55:23.819
of a little bit different of views than what you guys trying to achieve here because

00:55:23.819 --> 00:55:29.039
as a startup, now number one goal is to create a product, right? Ship it, right?

00:55:29.939 --> 00:55:34.979
So I cannot do all these kind of, you know, evil for the code agents.

00:55:36.479 --> 00:55:42.999
And when you're building a startup, I feel like all this coding agent for me is not an

00:55:42.999 --> 00:55:48.379
entry -level summer intern. Write some code, give them a test and give me a reading and

00:55:48.379 --> 00:55:48.719
greeting.

00:55:49.639 --> 00:55:53.319
I view them, each of these agents are like principle data managers or something like

00:55:53.319 --> 00:55:58.239
that behind. You know, kind of work for me, right? So when you can evaluate a principle

00:55:58.239 --> 00:56:04.739
engineer or whatever, I don't think we should pick the evil or give them

00:56:04.739 --> 00:56:07.399
a test, give them a greeting. We put them to compete,

00:56:08.459 --> 00:56:13.839
right? So I think this is kind of fundamental philosophy different than you do here.

00:56:14.319 --> 00:56:20.279
I feel like, you know, if I, as a startup guys, I will not using evil this way to kind

00:56:20.279 --> 00:56:26.319
of evaluate them. I will put them to compete with each other. I don't care who can get

00:56:26.319 --> 00:56:32.259
it done better, get cheaper or whatever, get it done, right? And I based on result to

00:56:32.259 --> 00:56:35.119
manage to pick which one will do more. For example, based on my experience, I use cloud

00:56:35.119 --> 00:56:40.179
innovating AI age. So I think you're absolutely right. If you make agents compete, the

00:56:40.179 --> 00:56:45.439
method by which you will tell which one is better is the evals bit. And you will never

00:56:45.439 --> 00:56:50.739
know who might have done it better or worse unless you run all of them. But if you run

00:56:50.739 --> 00:56:56.679
multiple, then you have to, you know, kind of assess their completion. That said, the

00:56:56.679 --> 00:57:00.599
big takeaway from this is not that I'm not telling everybody to go run their eval system.

00:57:00.819 --> 00:57:04.259
My big takeaway that I hope to convey is that be skeptical.

00:57:05.399 --> 00:57:10.759
What works as the best thing for a particular solution today may not be the best solution

00:57:10.759 --> 00:57:16.459
for that particular problem. And you see this, like if you look at open routers, traffic

00:57:16.459 --> 00:57:18.499
as well, like you'll see how traffic shifts

00:57:19.779 --> 00:57:24.379
between, you know, entropic and open AI models over time because people are just discovering

00:57:24.379 --> 00:57:28.999
that they're different. So remain skeptical, disconfirm your beliefs. Today, Codex might

00:57:28.999 --> 00:57:32.539
be really good at a particular task with Astra and tomorrow it might be the worst of

00:57:32.539 --> 00:57:38.859
the batch. As long as you can continue evolving your setup, it's probably fine. I think

00:57:38.859 --> 00:57:43.399
here it's more like one agent review others. For example, right? What I do is ask a cloud

00:57:43.399 --> 00:57:48.379
code writer spec. But now it's Codex to kind of review the spec and give me all the kind

00:57:48.379 --> 00:57:53.219
of hard feedback. And later I will do is maybe cloud, you can say incorporate all the

00:57:53.219 --> 00:57:58.739
feedback from Codex then kind of get a new version of the spec, right? So then kind of

00:57:58.739 --> 00:58:03.079
multiple version of this kind of work then you kind of, you will read the result at that

00:58:03.079 --> 00:58:07.499
time. You don't want to read at the very beginning, right? So then you have kind of decent

00:58:07.499 --> 00:58:12.619
spec to start to build something new. So that's kind of my experience. Using agents and

00:58:12.619 --> 00:58:16.799
adversarial capacity against each other is a really good idea. My workflow has that like

00:58:16.799 --> 00:58:20.319
automated as well. Like, so I think like using one to code review and having both of

00:58:20.319 --> 00:58:23.279
them code review. I think the most profound point of what you're saying

00:58:24.479 --> 00:58:29.079
is sometimes it's the output quality and like being able to one shot something is much

00:58:29.079 --> 00:58:35.559
more important than the dollars spent on it because your time matters a lot and that

00:58:35.559 --> 00:58:39.339
changes how much you can focus as well. We'll take one last question and then I think

00:58:39.339 --> 00:58:40.859
we'll be around after, but.

00:58:42.949 --> 00:58:45.489
Oh, sorry. I had one. Or we can take both.

00:58:47.909 --> 00:58:51.589
I was just curious, what's the, is you guys did all the different things you came up

00:58:51.589 --> 00:58:56.469
with. It was like the longest or hardest one. Like, were you, did you have any of your

00:58:56.469 --> 00:59:01.169
things that ran into the hours, multiple hours, multiple days that you'd have? Yeah,

00:59:01.169 --> 00:59:07.169
what's kind of the long end? Yeah, right now we're like, we're in the very, very early

00:59:07.169 --> 00:59:11.629
innings. So all our stuff is short horizon and one of the things that I called out is

00:59:11.629 --> 00:59:15.569
like a lot of the tasks that we have are not representative of a long horizon task, like

00:59:15.569 --> 00:59:19.509
a very diluted context history and so forth. So we haven't quite gotten that far. But

00:59:19.509 --> 00:59:23.049
like numbers, so like everything's right now under an hour. I take it the tasks that

00:59:23.049 --> 00:59:27.309
you're doing. Yeah, a couple hundred turns. No, no, no, it's about like, I think with

00:59:27.309 --> 00:59:32.289
grading and everything, it's around an hour because there's a suite, right? We do run

00:59:32.289 --> 00:59:37.629
public benchmarks. So like suite bench and terminal bench take multiple hours. Well,

00:59:37.769 --> 00:59:43.009
not the whole benchmark, the individual tasks. Yeah, so both suite bench and terminal

00:59:43.009 --> 00:59:49.069
bench have long horizon tasks that take multiple hours. We haven't come up with our own

00:59:49.069 --> 00:59:55.449
that take multiple hours, yeah. But it's one of the things that I think even in the research

00:59:55.449 --> 00:59:59.949
community is being talked about a lot because if you think about long horizon tasks in

00:59:59.949 --> 01:00:06.009
real life, rarely do they never require human input. But that's what these long horizon

01:00:06.009 --> 01:00:11.209
tasks are is that they're isolated and they require zero human input. I've had a few.

01:00:15.409 --> 01:00:15.669
Yep.

01:00:19.449 --> 01:00:25.949
This thing is, it's evaluating the agent itself, not the model, right? Meaning like,

01:00:25.969 --> 01:00:32.489
for example, when you use co -pilot and it routes different kinds of

01:00:32.489 --> 01:00:39.269
tasks to different kinds of models, then it can get the cheap tasks to the cheap models

01:00:39.269 --> 01:00:44.549
and the more expensive tasks and the more expensive models. Is that correct? We haven't

01:00:44.549 --> 01:00:48.889
gotten quite that sophisticated. We've built our own routing infrastructure and so forth,

01:00:48.949 --> 01:00:53.689
but we haven't tested. There's a whole lot of work to be done here. Testing what happens

01:00:53.689 --> 01:00:57.749
to different skills, what happens to different routing philosophies. There's a whole

01:00:57.749 --> 01:01:03.089
lot. We're just kind of getting started. But here, we did a very controlled set of experiments

01:01:03.089 --> 01:01:08.929
on one -shotting or surveying some long horizon tasks against a specific model with a

01:01:08.929 --> 01:01:11.889
specific harness at a specific reasoning level.

01:01:13.009 --> 01:01:18.689
All right, and then one thing I was really interested in is have you had much success

01:01:18.689 --> 01:01:24.689
testing things that are more fuzzy, like readability? For example, if you have an agent

01:01:24.689 --> 01:01:30.969
write one function that's 500 lines, that's probably bad. But also, if you have it write

01:01:30.969 --> 01:01:34.209
50 functions that are five lines, that's also probably really bad.

01:01:35.669 --> 01:01:41.289
That goes into the representative versus measurable part as well, because readability

01:01:41.289 --> 01:01:42.629
is also very, very subjective.

01:01:43.729 --> 01:01:48.649
You often have, as I mentioned, that one engineer that has very specific kind of taste.

01:01:49.429 --> 01:01:54.769
And every organization has these different standards. Now, you can run linter rules and

01:01:54.769 --> 01:01:58.949
so forth. That's very, very measurable, but that's where you start getting into LLM as

01:01:58.949 --> 01:02:05.809
a judge. To be able to assess how well this does compared to the essence of the Git repository

01:02:05.809 --> 01:02:10.889
as well. In the evaluation framework that we have right now, we do include an LLM judge

01:02:10.889 --> 01:02:16.249
that looks at conventions and readability and maintainability. A lot of that is just

01:02:16.249 --> 01:02:21.009
regular coding standards. What we really wanna be able to do is actually scan your repo

01:02:21.009 --> 01:02:25.689
to see what you prefer as an organization. There's also other things like risk tolerance,

01:02:26.109 --> 01:02:32.769
right? At certain organizations, you tolerate less code that can sort of

01:02:32.769 --> 01:02:38.009
fail in unexpected ways, like throw an error. That's an unclot exception, things like

01:02:38.009 --> 01:02:43.889
that. And if you've ever used this skill inside Clot Code, Clot Convention Review, it's

01:02:43.889 --> 01:02:48.789
very similar. It's basically like a convention review judge that you apply in addition

01:02:48.789 --> 01:02:52.929
to the test passing, which is correctness, you have the subjective test that's basically

01:02:52.929 --> 01:02:57.729
the LLM that's not like a random LLM. You have to prompt the LLM to check for specific

01:02:57.729 --> 01:03:02.429
things. Thank you for having us. And thank you Antithesis and Chris for sponsoring.

01:03:04.129 --> 01:03:05.169
Thank you.

01:03:06.329 --> 01:03:06.369
Yeah, I have one quick question, Mark, for you. Someone asked, your slide said