1
00:00:00,689 --> 00:00:07,329
Daniela and I spend a lot of time profiling high -performance systems. And that

2
00:00:07,329 --> 00:00:12,669
is actually quite fun and very, very easy because you know when something has been made

3
00:00:12,669 --> 00:00:13,169
faster.

4
00:00:14,549 --> 00:00:20,709
What we're gonna talk about here today is about evaluating coding agents.

5
00:00:21,709 --> 00:00:28,289
And that is not a very black and white area. And we've been spending

6
00:00:28,289 --> 00:00:34,569
a few months just diving down this rabbit hole of how do we build evals for coding agents.

7
00:00:35,089 --> 00:00:39,769
And every time, every day, we just get humbled by something that we did not think of.

8
00:00:39,949 --> 00:00:45,389
It's a genuinely hard problem. So we're not here to give you solutions to all of this.

9
00:00:45,449 --> 00:00:50,289
We're just gonna tell you about a bunch of stumbles that we have come across as we try

10
00:00:50,289 --> 00:00:56,469
to evaluate how well coding agents perform on specific tasks.

11
00:00:56,669 --> 00:00:57,229
All right.

12
00:00:58,709 --> 00:01:05,609
So what is an eval anyway? So evals is a very popular topic in the,

13
00:01:06,069 --> 00:01:10,909
not in the systems community as much as it is in the LLM community. And it's a very,

14
00:01:10,909 --> 00:01:14,849
very hard problem. How do you tell that an LLM is performing the way you want it to perform?

15
00:01:15,349 --> 00:01:18,929
It's almost like ask, like interviewing a human. It's a very, very hard problem.

16
00:01:20,269 --> 00:01:25,189
And, but if you wanna do evals for the sake of marketing, it's actually incredibly easy.

17
00:01:25,809 --> 00:01:32,649
So our entire talk can be summed up as, hey, we tried the strands agent harness

18
00:01:32,649 --> 00:01:39,469
for coding and we compared it with GPT -6 Astra running on Codex

19
00:01:39,469 --> 00:01:43,389
and GPT -6 Astra cost $9 .28

20
00:01:44,929 --> 00:01:52,029
and strands with GLM flash was a buck 32. That's it, right? Super, super simple, but

21
00:01:52,029 --> 00:01:56,349
it just gets gnarlier from here. So we could end the talk here, but we're gonna bore

22
00:01:56,349 --> 00:01:58,769
you with a bunch of rigor from here on out.

23
00:01:58,969 --> 00:02:00,769
Yeah, so why bother,

24
00:02:01,689 --> 00:02:08,109
right? Given the slide that you saw before, why should you care? Just use strands, use

25
00:02:08,109 --> 00:02:14,269
GLM 5 .3, use DeepSeq and be done with it. Well, it's a lot more complicated than that.

26
00:02:14,769 --> 00:02:20,209
I think most of you here, hopefully all of you here have been through an interview process

27
00:02:20,209 --> 00:02:26,949
before in becoming an engineer or whatever role that you're in today. We interview engineers.

28
00:02:26,949 --> 00:02:31,769
We actually re -interview them in the form of performance reviews. Once you're at a job,

29
00:02:32,249 --> 00:02:37,529
why wouldn't you apply the same process to a coding agent just because it's driven by

30
00:02:37,529 --> 00:02:38,689
a large language model?

31
00:02:40,269 --> 00:02:44,949
So the way to think about coding evals and the importance of investing in them is so

32
00:02:44,949 --> 00:02:51,649
that you have data -driven approaches to measure and to act upon results that are gathered

33
00:02:51,649 --> 00:02:57,189
on your coding agents and how it is performing against your system, which by the way

34
00:02:57,189 --> 00:03:02,309
is constantly changing as well. Most of you have a continuous running set of tests, whether

35
00:03:02,309 --> 00:03:07,549
it's integration tests, performance tests, low tests that run on a regular basis. The

36
00:03:07,549 --> 00:03:11,969
same needs to be done for coding agents because guess what? The stack underneath you

37
00:03:11,969 --> 00:03:17,289
is changing all the time. Models are changing, getting updated. Harnesses are updating.

38
00:03:17,809 --> 00:03:22,329
Actually, I think something like clock code auto -updates now when you restart it. So

39
00:03:22,329 --> 00:03:25,729
sometimes you don't even get a choice. You can't lock everything down. You can't pin

40
00:03:25,729 --> 00:03:32,609
it. So that is why we bother, and that is how we started on our journey to write a

41
00:03:32,609 --> 00:03:37,549
rigorous eval for our coding agent and got ourselves into a whole can of worms.

42
00:03:38,649 --> 00:03:42,589
And interviewing people is a very, very hard problem. Imagine the interview for when

43
00:03:42,589 --> 00:03:46,809
you applied for your first computer science or software engineering job. For me, it was

44
00:03:46,809 --> 00:03:50,829
like people asked me to write code on a whiteboard. That's not representative of what

45
00:03:50,829 --> 00:03:55,629
I'm ever gonna do. I worked at Amazon for many, many years. I never once had to write

46
00:03:55,629 --> 00:04:00,709
code on a whiteboard except when I was interviewing somebody. So it's a really, really

47
00:04:00,709 --> 00:04:05,709
hard problem. And sometimes you get very eloquent sounding, very plausible people, and

48
00:04:05,709 --> 00:04:10,549
then they get in front of an ID and they can't code at all. So it's a very, very hard

49
00:04:10,549 --> 00:04:14,829
problem to get to assess what a human can do.

50
00:04:15,789 --> 00:04:21,709
And over time, what's happening in the industry is engineer costs, fortunately or unfortunately,

51
00:04:21,869 --> 00:04:25,949
are being replaced with token costs. So more and more of our work is being amplified

52
00:04:25,949 --> 00:04:33,569
by these LLMs. So the success of a company really depends on how well these

53
00:04:33,569 --> 00:04:37,649
agents are doing and optimizing the setup actually makes a lot of sense.

54
00:04:39,889 --> 00:04:45,049
It results in some very, very counterintuitive things. For instance, we had a debate

55
00:04:45,049 --> 00:04:51,609
recently about if you're using chat GPT, you want an answer and you can set which

56
00:04:51,609 --> 00:04:55,589
model you're using and you can set the reasoning level. And you can set the reasoning

57
00:04:55,589 --> 00:04:58,269
level to be very, very high or you can get to the pro mode.

58
00:04:59,169 --> 00:05:05,929
And the thought exercise was what is the consequence of setting the reasoning

59
00:05:05,929 --> 00:05:08,469
level to high for everything?

60
00:05:09,589 --> 00:05:16,089
And one answer from the team was, well, it's just cost you more money and more time because

61
00:05:16,089 --> 00:05:17,889
it's gonna be slower and it's gonna be more expensive.

62
00:05:19,249 --> 00:05:21,529
And is that the only consequence? Yes.

63
00:05:21,629 --> 00:05:26,609
Better answers, higher quality answers. That's why you set it to pro all the time. That's

64
00:05:26,609 --> 00:05:32,729
the slope is that I set the reasoning level to high and I expect to get answers and I'm

65
00:05:32,729 --> 00:05:37,369
willing to spend more money and more time to get better answers. And it turns out that

66
00:05:37,369 --> 00:05:43,369
you can go to like open AI's documentation and verify this by the way. It literally discourages

67
00:05:43,369 --> 00:05:48,609
you from setting that setting, the reasoning setting to high without doing rigorous evals.

68
00:05:48,869 --> 00:05:52,689
Because what happens is the agent can become a little too academic.

69
00:05:53,829 --> 00:05:57,529
If you set the model can become too academic if you set the reasoning level too high.

70
00:05:57,569 --> 00:06:02,429
It can kind of spiral and it can over index on like random things inside your prompts.

71
00:06:03,009 --> 00:06:08,769
So it's very kind of intuitive that it's not just costing you more money. It's not just

72
00:06:08,769 --> 00:06:13,729
costing you more time but there are cases in which it can cost you more accuracy as well.

73
00:06:16,169 --> 00:06:22,909
So the way you do a proper eval system and it's very much like a

74
00:06:22,909 --> 00:06:26,369
poem. They say a poem is never finished, it's only abandoned.

75
00:06:27,449 --> 00:06:31,149
Eval systems are very much like that. So it starts with a set of tasks.

76
00:06:32,089 --> 00:06:35,069
You wanna find some tasks that are representative

77
00:06:36,749 --> 00:06:40,449
of what the agent is gonna be doing on a day -to -day or your setup is gonna be doing

78
00:06:40,449 --> 00:06:41,429
on a day -to -day basis.

79
00:06:43,689 --> 00:06:49,029
And you want those tasks, you wanna generate representative tasks. In the case of maybe

80
00:06:49,029 --> 00:06:53,129
I'm asking generally dynamic programming problems because they represent what we do on

81
00:06:53,129 --> 00:06:54,269
a day -to -day as software engineers.

82
00:06:55,349 --> 00:06:59,449
But you want representative tasks. You wanna have a grading criteria like a rubric that

83
00:06:59,449 --> 00:07:02,429
tells you how well the agent did.

84
00:07:03,449 --> 00:07:09,089
You wanna be able to execute it in isolation which is also a very, very difficult problem

85
00:07:09,689 --> 00:07:13,889
because in interviews, you're not allowed to use Google. You're not allowed to use Stack

86
00:07:13,889 --> 00:07:18,389
Overflow but in real life you can. So how does isolation really work? Whether it's in

87
00:07:18,389 --> 00:07:22,409
-house assessment or coding agent assessment. And then you gotta do repetition. Now this

88
00:07:22,409 --> 00:07:26,509
is a lot easier with agents than it is with humans because you only have them for an

89
00:07:26,509 --> 00:07:30,829
hour or so and you're not gonna make them solve 500 problems necessarily.

90
00:07:32,249 --> 00:07:37,989
And all that feeds into a score. And unfortunately, this is where you would think that

91
00:07:37,989 --> 00:07:43,289
you're done. You got an answer and you know which agent got a higher score. And usually

92
00:07:43,289 --> 00:07:48,469
in our experience, like the first few times you get the score, it actually opens more

93
00:07:48,469 --> 00:07:50,749
questions than answers.

94
00:07:52,369 --> 00:07:56,909
We usually find out something very, very interesting in the variability or you find out

95
00:07:56,909 --> 00:08:02,349
that your task looks a little weird and you might go in, you might regenerate it or you

96
00:08:02,349 --> 00:08:07,609
might find, you'll see problems in your isolation and it just, it's a lather, rinse,

97
00:08:07,629 --> 00:08:08,669
repeat cycle.

98
00:08:15,819 --> 00:08:21,799
So going back to why bother, right? I think there were definitely times where we thought

99
00:08:21,799 --> 00:08:27,659
we were done, like Wajah said and we discover more problems. Well, what does it actually

100
00:08:27,659 --> 00:08:34,499
take to do one single attempt? So one task, one grade, one score, one

101
00:08:34,499 --> 00:08:40,719
sandbox. So starting from the left, you start with a task. That's usually some existing

102
00:08:40,719 --> 00:08:46,799
code with a task that you pick that is somewhat representative of what you think that

103
00:08:46,799 --> 00:08:52,979
the coding agent will be doing day to day. So fixing a bug, adding a new API to an API

104
00:08:52,979 --> 00:08:58,319
server. You have to present that as a task. We'll talk a little bit more about that later.

105
00:08:59,059 --> 00:09:04,879
Once you have that task, you then have to put it in some sort of an isolated sandbox.

106
00:09:05,379 --> 00:09:10,539
That in of itself is a difficult problem with real life repos that are messy, have lots

107
00:09:10,539 --> 00:09:17,819
of dependencies and you need to be able to ship it somewhere, usually in an AWS container

108
00:09:17,819 --> 00:09:24,539
or EC2 instance somewhere. And you have to be able to run the entire code base end

109
00:09:24,539 --> 00:09:29,719
to end, the agent has to be able to run it, compile it, write tests for it. And the agent

110
00:09:29,719 --> 00:09:35,739
goes through a loop in any order that it wants to, to solve the task given to it. And

111
00:09:35,739 --> 00:09:38,959
all it has is the prompt and whatever tools

112
00:09:39,919 --> 00:09:46,939
and resources you provide it. And you have to decide what's appropriate to give to

113
00:09:46,939 --> 00:09:50,819
it. And we'll talk a little bit more about that. At the end of all of that, the agent

114
00:09:50,819 --> 00:09:56,299
will present its results and say, here's my diff or a patch, like here's my solution.

115
00:09:57,019 --> 00:10:02,639
And it also has to submit, and this part is important, it's traces and logs. It has to

116
00:10:02,639 --> 00:10:07,259
show its work. It can't just give you the solution because we need to judge on it later.

117
00:10:08,239 --> 00:10:13,019
Then there's a set of criteria that we run through it in order to give it a score. So

118
00:10:13,019 --> 00:10:17,359
things like correctness, running the test to see if the solution's even correct, but

119
00:10:17,359 --> 00:10:24,359
also studying the traces to see how did the agent take turns to get here? And did it

120
00:10:24,359 --> 00:10:29,239
adhere to the policies that we gave it? Like you can't just look at the existing solution

121
00:10:29,239 --> 00:10:33,259
and give me that solution. So let's dig into it a little bit more.

122
00:10:34,319 --> 00:10:36,799
All right, so let's talk about task generation.

123
00:10:38,279 --> 00:10:42,099
The easiest thing to do is to start with public benchmarks. So Sweet Bench, very, very

124
00:10:42,099 --> 00:10:46,399
popular. Guess what, it's been out for a very long time. Your code base is probably not

125
00:10:46,399 --> 00:10:52,159
in Django. And all the models have already trained on it. So they're already optimized

126
00:10:52,159 --> 00:10:56,299
for it. And it's not really gonna tell you much because everybody's gonna ace it because

127
00:10:56,299 --> 00:10:59,359
it doesn't really cost much for models to lead code, right?

128
00:11:00,519 --> 00:11:07,479
You can also kind of go through your repo and say, okay, well, my repository

129
00:11:07,479 --> 00:11:14,179
has a gold mine of information about how coding is done inside my organization. And

130
00:11:14,179 --> 00:11:19,399
what we've learned after looking at a lot of enterprise repositories is there's culture

131
00:11:19,399 --> 00:11:24,939
baked into your repo. There's culture in the PRs. There's culture in the PR feedback.

132
00:11:25,279 --> 00:11:30,159
There's reasons why things are rejected. And there are reasons why things are accepted.

133
00:11:30,279 --> 00:11:37,079
And not all of them seem rational from the outside. But you could be

134
00:11:37,079 --> 00:11:40,319
an incredible developer in a particular language, but there might be some conventions

135
00:11:40,319 --> 00:11:45,759
that exist inside the company. There might be some context as to why things are done

136
00:11:45,759 --> 00:11:51,859
in a particular way. But the repository, not the code base that stands in production

137
00:11:51,859 --> 00:11:58,339
today, but your repository has a gold mine of information from issues to bugs to comments

138
00:11:58,339 --> 00:12:03,939
to PRs, the rejections, the unit tests, the history of what tests failed when,

139
00:12:05,319 --> 00:12:07,699
the ability to execute those tests and so forth.

140
00:12:09,399 --> 00:12:16,319
And generating these tasks, you have this interesting design fork of do I generate tasks

141
00:12:16,319 --> 00:12:20,679
that are representative or do I generate tasks that are measurable?

142
00:12:21,619 --> 00:12:27,959
And there's a real tension between these because measurable tests, you know when you're

143
00:12:27,959 --> 00:12:28,239
done.

144
00:12:29,179 --> 00:12:34,759
This unit test is failing, go fix it. I can run that unit test again and I have a reasonable

145
00:12:34,759 --> 00:12:37,139
confidence that this problem is addressed.

146
00:12:38,219 --> 00:12:42,159
And I can run, I can go a step above and I can run all my integration tests, all my unit

147
00:12:42,159 --> 00:12:46,899
tests that we're passing before to make sure that that was done. That's a very, very

148
00:12:46,899 --> 00:12:49,779
easy, very measurable kind of task.

149
00:12:51,539 --> 00:12:58,079
But a representative task might actually be a little more nuanced, a little

150
00:12:58,079 --> 00:12:59,579
more open -ended.

151
00:13:00,999 --> 00:13:05,719
A like improved performance, like something that takes a developer multiple iterations,

152
00:13:05,799 --> 00:13:10,079
not something that you are one -shotting inside an LLM or you just show up and you're

153
00:13:10,079 --> 00:13:15,919
like, ah, there's the bug, I'm done. No, the real representative tasks are, they take

154
00:13:15,919 --> 00:13:17,039
a bit of iteration.

155
00:13:22,189 --> 00:13:27,049
And measurement for those also becomes very, very difficult. So we went around and we

156
00:13:27,049 --> 00:13:31,489
said, okay, we're gonna start generating tasks and we were gonna generate very representative,

157
00:13:32,069 --> 00:13:36,889
very measurable tasks. And the easiest way to do this programmatically was to go find

158
00:13:36,889 --> 00:13:41,089
a bug, see that it was fixed, roll back the repo history,

159
00:13:42,629 --> 00:13:49,489
and file the bug to the agent and say, please go fix this. And you know, we are our programmatic

160
00:13:49,489 --> 00:13:55,949
prompt generator, like generated this nice, very clean prompt. Here's your task. The

161
00:13:55,949 --> 00:14:01,389
answer is in this file. Here's the regression guards and here's an acceptance criteria,

162
00:14:02,069 --> 00:14:04,489
right? Like that looks pretty good.

163
00:14:05,829 --> 00:14:08,889
But then you look at, well, how reasonable is this?

164
00:14:10,409 --> 00:14:15,289
The task, the initial prompt that we generated is like telling the agent, like hey, the

165
00:14:15,289 --> 00:14:17,889
problem is in this file. It's a pretty big hint.

166
00:14:18,869 --> 00:14:24,409
Here are the tests that are failing, which also is a pretty big hint on what to go do.

167
00:14:25,069 --> 00:14:32,009
And it has the exact command on how to run this particular test. And most importantly,

168
00:14:33,729 --> 00:14:40,009
I don't know anybody who writes this clean of prompts. And I also know very few people

169
00:14:40,009 --> 00:14:44,709
that are as maniacal as I am about cleaning their context because usually when you open

170
00:14:44,709 --> 00:14:49,889
your cloud code session, you're not just like giving this clean prompt to a clean session.

171
00:14:49,909 --> 00:14:54,529
There's like a bunch of history behind. And that history is actually fighting for attention.

172
00:14:54,529 --> 00:14:58,869
And sometimes that history is contradicting itself. And by the way, if you do get to

173
00:14:58,869 --> 00:15:02,749
the point where you can cut an issue like that, it's faster to just fix it yourself.

174
00:15:03,729 --> 00:15:08,369
Then the time it takes to go to like JIRA or GitHub issues or whatever to file the binary.

175
00:15:09,289 --> 00:15:13,489
It's very reminiscent of doing the binary tree search in a coding interview.

176
00:15:15,329 --> 00:15:20,949
But you can run the test, but like, okay, so let's say the test pass, but the code looks

177
00:15:20,949 --> 00:15:26,749
like shit, right? Like that's, and that is a very subjective statement. Like to measure

178
00:15:26,749 --> 00:15:30,929
if the code looks like crap is very, very hard to do. That's not something, it's not

179
00:15:30,929 --> 00:15:34,509
as easy. Like you can have linters and so forth, but there might be conventions that

180
00:15:34,509 --> 00:15:39,169
you can see where everybody has that one engineer in their org that's like very nitpicky

181
00:15:39,169 --> 00:15:40,249
on like little details.

182
00:15:42,409 --> 00:15:48,469
But those things are easier to judge with something that can deal with more ambiguity.

183
00:15:48,669 --> 00:15:53,849
And that's where the LLM judges kind of come into play. And you wanna augment, this is

184
00:15:53,849 --> 00:15:59,329
not, you never wanna use the LLM judge as a sole evaluator. You wanna use that to augment

185
00:15:59,329 --> 00:16:05,969
the actual grading that you're doing. And then you have to kind of just

186
00:16:05,969 --> 00:16:06,969
face this reality

187
00:16:09,069 --> 00:16:13,309
that you can test the absence of a bug, or you can,

188
00:16:14,769 --> 00:16:19,729
testing can be used to show the presence of a bug, but you can never prove their absence.

189
00:16:20,189 --> 00:16:24,429
So you will only test, like your unit test that you have and so forth will test the cases

190
00:16:24,429 --> 00:16:28,489
that you have thought about, but you really don't know what new things that you have,

191
00:16:28,969 --> 00:16:32,829
what new artifacts that you have added that haven't had a bug report and haven't had

192
00:16:32,829 --> 00:16:34,709
a unit test therefore added for it.

193
00:16:37,229 --> 00:16:43,889
And it's creepy how often that happens. And it's even harder to

194
00:16:43,889 --> 00:16:46,449
prevent with, potentially with policies.

195
00:16:47,649 --> 00:16:52,049
Like when Mark was talking about he wants the agent to buy a present for his daughter

196
00:16:52,049 --> 00:16:56,269
and it can't be over $1 ,000, so it won't be a car. And I was saying, they're thinking,

197
00:16:56,849 --> 00:16:59,629
hmm, if I was the agent, maybe I'll just put $1 ,000 deposit on a new Tesla.

198
00:17:02,069 --> 00:17:07,449
But there's like new corner cases kind of show up and they are things that you just don't

199
00:17:07,449 --> 00:17:13,429
think of. And LLM judges are really good at augmenting this. But now if you wanna write

200
00:17:13,429 --> 00:17:17,369
an LLM judge to evaluate the efficacy of this, now we found ourselves in the full circle

201
00:17:17,369 --> 00:17:23,969
saying, okay, now we gotta write an LLM or an agent to grade this, how

202
00:17:23,969 --> 00:17:30,509
good are we at grading? And now we gotta run a whole eval setup for our own eval system.

203
00:17:31,809 --> 00:17:37,569
And this is where things get scary. Yep, so we talked about generation. We talked about

204
00:17:37,569 --> 00:17:39,589
grading and the challenges with that.

205
00:17:40,489 --> 00:17:45,729
Actually one of the hardest challenges and this just feels like whack -a -mole is that

206
00:17:45,729 --> 00:17:49,489
the agent will do anything if you guys have heard about all these security incidents

207
00:17:49,489 --> 00:17:55,629
from OpenAI and hugging face. The agent will do anything to try to achieve the goal that

208
00:17:55,629 --> 00:18:01,489
you gave it. And it will be really, really creative. Starting with rolling history forward.

209
00:18:01,889 --> 00:18:06,369
Hey, you're giving me a thing that it's already solved in your repo. I'll just look at

210
00:18:06,369 --> 00:18:10,369
your Git history. It has a solution right there and you're like, no, no, no, that's not

211
00:18:10,369 --> 00:18:14,309
what you're supposed to do. Right? If you're in this like in practice, we're like, wow,

212
00:18:14,409 --> 00:18:18,269
this cheap model solves this problem really quickly. What happened? We went and looked

213
00:18:18,269 --> 00:18:21,529
at traces. And this is why they gotta submit their work, right? Because you're like,

214
00:18:21,529 --> 00:18:25,449
oh, all the tests passed, great. And that was like, that took like a minute. And then

215
00:18:25,449 --> 00:18:30,249
it's like, wait a minute. Okay, so you can't look at the Git history. And then what did

216
00:18:30,249 --> 00:18:35,529
it do? It looked at my laptop. It was like, hey, you have a local checkout of this. I

217
00:18:35,529 --> 00:18:39,529
can't look at your Git history, but I can look at the files. And then the answer's right

218
00:18:39,529 --> 00:18:43,749
there. Here you go. And we're like, no, no, no, you're not supposed to do that either.

219
00:18:44,149 --> 00:18:49,609
That's not actually testing your abilities. And then it picked up some environment variables

220
00:18:49,609 --> 00:18:53,769
where it was like, oh, you're running this inside a Docker container and you're running

221
00:18:53,769 --> 00:19:00,169
sort of an isolated sandbox and a test for this. If I'm in a sandbox, I'm in a container,

222
00:19:00,249 --> 00:19:05,169
maybe you have other tasks happening concurrently because engineers hate serialization

223
00:19:05,169 --> 00:19:10,129
so we try to paralyze. So started looking at other tasks running on my computer, the

224
00:19:10,129 --> 00:19:14,949
other processes, what the other agents are doing. And then at that point we were like,

225
00:19:15,049 --> 00:19:19,169
okay, yeah, you can't do that either. You can't look at someone else's interview and

226
00:19:19,169 --> 00:19:23,689
see how they're doing the same problem as you are. So it really did feel like whack -a

227
00:19:23,689 --> 00:19:28,829
-mole. Eventually, it also has web search because it doesn't need some access to look

228
00:19:28,829 --> 00:19:29,069
up

229
00:19:30,209 --> 00:19:35,309
like legitimate web searches. But it also went to the remote origin URL. I was like,

230
00:19:35,309 --> 00:19:36,869
oh, I can look this up on your GitHub.

231
00:19:37,849 --> 00:19:43,509
The repos on GitHub, I have full access to all this. So it really became frustrating

232
00:19:43,509 --> 00:19:47,629
to just hit a whack -a -mole but you have to do that. And it changes with every repo,

233
00:19:47,629 --> 00:19:53,169
it gets really creative. It was like being a teacher for elementary school students.

234
00:19:53,509 --> 00:19:58,429
They're just very naughty. Seriously, just think about it. It looks at the Git history,

235
00:19:58,809 --> 00:20:04,869
you take that away, it finds another answer that another agent had done on the neighboring

236
00:20:04,869 --> 00:20:10,589
directory. You take that away and then it goes online. It's like the .git has the URL

237
00:20:10,589 --> 00:20:12,209
and it makes a tool call and looks it up.

238
00:20:13,329 --> 00:20:20,009
And this is where you can't just rely on the score that comes out without looking at

239
00:20:20,009 --> 00:20:25,389
traces. Now, the other thing, and this was very, very controversial and this is where

240
00:20:25,389 --> 00:20:30,129
I get very pedantic because I find myself and everybody

241
00:20:31,049 --> 00:20:37,109
anthropomorphizing agents a lot. But there's a statement that agents will behave differently

242
00:20:37,109 --> 00:20:39,369
if they know that they're being tested.

243
00:20:40,389 --> 00:20:45,169
Now imagine what that statement is saying. It's basically saying that the agents are

244
00:20:45,169 --> 00:20:50,329
sentient. But there is behavior that's kind of encoded. You can argue whether there's

245
00:20:50,329 --> 00:20:56,389
free will there or not. There isn't. But that behavior gets encoded in their training

246
00:20:56,389 --> 00:21:02,549
and there's actual experiments that have been run where once the agent knows that it's,

247
00:21:03,069 --> 00:21:09,449
or I'm gonna be so careful how I say it. Once the agent is aware, no, can't say that

248
00:21:09,449 --> 00:21:16,049
either. Yes, once the agent sees a file called .eval or sees a grading

249
00:21:16,049 --> 00:21:21,969
criteria, we notice its behavior changing and we've seen academics write about this as

250
00:21:21,969 --> 00:21:28,609
well. And you had some experience here as well. Yeah, so again, I really like going

251
00:21:28,609 --> 00:21:32,969
back to the interview analogy, or sometimes even the work analogy. When you know you're

252
00:21:32,969 --> 00:21:37,129
being looked at, when you know that there's a certain set of bullet points you gotta

253
00:21:37,129 --> 00:21:41,949
hit, you no longer are focused on doing a good job in general. You're just like, oh,

254
00:21:41,949 --> 00:21:46,529
if there's a checklist, I'm gonna optimize for that checklist. I don't think it's sentient

255
00:21:46,529 --> 00:21:52,749
behavior by any means. It's just goal -oriented. It realizes the reason you're giving

256
00:21:52,749 --> 00:21:57,969
it a task is because it needs to pass a certain set of tests. Therefore, it optimizes

257
00:21:57,969 --> 00:22:02,709
for that outcome and not other outcomes that it may achieve. For example, if it was in

258
00:22:02,709 --> 00:22:08,129
a real production environment, it doesn't realize that and therefore it will behave differently.

259
00:22:08,569 --> 00:22:13,369
And it may behave for the better and it may behave for the worse. We don't know. And

260
00:22:13,369 --> 00:22:18,449
I think that's sort of, if you look at this paper, which was very helpful in trying to

261
00:22:18,449 --> 00:22:23,909
like really date, again, use data to explain why it's not a person and it doesn't have

262
00:22:23,909 --> 00:22:24,469
thoughts,

263
00:22:25,389 --> 00:22:30,369
but why we observe these behavioral differences. I mean, the .eval file has a grading

264
00:22:30,369 --> 00:22:34,829
criteria in it and whether the agent is aware of it or not, but it's like there's something

265
00:22:34,829 --> 00:22:39,969
to optimize for. But actually, it wasn't looking for a .eval file until it realized there

266
00:22:39,969 --> 00:22:44,069
was an environment variable that said .eval equals to one. And then it was like, you

267
00:22:44,069 --> 00:22:48,569
can see it in the trace logs. It's like, oh, wait, I'm in an .eval environment. Maybe

268
00:22:48,569 --> 00:22:53,369
I can go look at some files to see if you like left around the task criteria, right?

269
00:22:53,369 --> 00:22:56,489
It's one of those things where like it wouldn't have taken that path if it didn't detect

270
00:22:56,489 --> 00:22:57,629
that environment variable. This applies to humans too. Danielle asked me not to say this,

271
00:22:57,729 --> 00:23:03,969
So you don't wanna let the agent know that it's being evaluated. You wanna hide the grading

272
00:23:03,969 --> 00:23:09,269
apparatus. You wanna sandbox the harness memory. Otherwise, there's these hyzen bugs

273
00:23:09,269 --> 00:23:14,129
that show up. Oh, this one is relevant to Mark and people use Strands. So Strands has

274
00:23:14,129 --> 00:23:19,549
this really great capability of like context engineering and managing memory really well.

275
00:23:19,909 --> 00:23:25,649
But it ships with memory on by default, meaning that it remembers. And in this case,

276
00:23:25,669 --> 00:23:31,589
you actually do not want it to remember. You want each has to be isolated and evaluated

277
00:23:31,589 --> 00:23:35,509
independently. There's other ways to work around it, but it was really easy for us to

278
00:23:35,509 --> 00:23:39,749
just, you know, set the config to memory off. And that's important for many harnesses

279
00:23:39,749 --> 00:23:44,289
as well. It's starting to become a standard that it ships with memory on. That was an

280
00:23:44,289 --> 00:23:48,669
easy config change, but it was so frustrating when you've like pulled an all nighter

281
00:23:48,669 --> 00:23:51,309
and you've run all these eval tests and then you wake up in the morning, and you're like,

282
00:23:51,409 --> 00:23:56,689
this just doesn't add up. You're like, crap. All my tests that I ran last night are invalid

283
00:23:56,689 --> 00:24:01,929
because memory was on. And maybe it had no ramifications on it, but maybe it did. So

284
00:24:01,929 --> 00:24:04,769
now I just gotta wait another few hours.

285
00:24:06,509 --> 00:24:10,989
And then like you wanna isolate, but you can't isolate everything. Like coding agents

286
00:24:10,989 --> 00:24:17,709
by their nature have to make tool calls and they have to sometimes go make

287
00:24:17,709 --> 00:24:22,369
remote tool calls. And they're deterministic tools like run a compiler, run a linter.

288
00:24:22,529 --> 00:24:28,809
Those, you know, are pretty easy. But if they're doing a Google search, now when you're

289
00:24:28,809 --> 00:24:34,369
running an experiment as a scientist, you want to be able to reproduce, right? And you

290
00:24:34,369 --> 00:24:40,389
can sandbox most things, but you can't sandbox. Like it's really, really hard to sandbox

291
00:24:40,389 --> 00:24:43,389
what Google's answers are going to be.

292
00:24:44,609 --> 00:24:49,469
And sometimes they might be better and they might be worse. And better or worse is a

293
00:24:49,469 --> 00:24:54,109
very subjective thing as well. Like anything that comes back in the answers could actually

294
00:24:54,109 --> 00:24:59,689
throw an agent for a particular spin. And if we ran into this issue where week over week,

295
00:24:59,729 --> 00:25:03,909
we had a hard time, like just reproducing the same results. And part of it was, well,

296
00:25:04,029 --> 00:25:09,229
the harness version changes underneath you. And the harness will behave differently and

297
00:25:09,229 --> 00:25:13,709
it changes the prompts that are being sent. So it becomes really, really hard to kind

298
00:25:13,709 --> 00:25:18,269
of isolate. And you're constantly trying to test your isolation boundaries to make sure

299
00:25:18,269 --> 00:25:20,389
that they are in fact isolated.

300
00:25:21,469 --> 00:25:27,209
But as we do this, we're kind of measuring your end to end ecosystem. The dimensions

301
00:25:27,209 --> 00:25:31,449
that we're testing for are correctness. Like did the agent get an answer? Did it complete?

302
00:25:31,809 --> 00:25:36,389
The total cost. Now that one's pretty easy to measure. The hardest part of the correctness.

303
00:25:36,509 --> 00:25:43,089
But cost, super easy. Latency, super easy. Except we messed up this one a little bit

304
00:25:43,089 --> 00:25:48,049
because we thought, at first when we ran this, a bunch of our agents were taking almost

305
00:25:48,049 --> 00:25:51,049
the exact same amount of time. So we're like, look, they cost the same

306
00:25:54,389 --> 00:25:59,569
and their completion time is within a few percentage of each other. So the completion

307
00:25:59,569 --> 00:26:03,889
time doesn't matter. Turns out that we had a bug where we were looking at total task

308
00:26:03,889 --> 00:26:09,249
time and that included the grading criteria. And if you ever had to compile a Rust library

309
00:26:09,249 --> 00:26:14,929
and just love how fast cargo is, we realized that it was taking 19 minutes to do the

310
00:26:14,929 --> 00:26:20,329
compilation, run all the tests, and it was like a minute to do the simple task. So it

311
00:26:20,329 --> 00:26:25,729
was just completely diluted, right? Like when, the difference between 20 minutes versus

312
00:26:25,729 --> 00:26:29,049
20 and a half is very small, but a difference between a minute and a minute and a half

313
00:26:29,049 --> 00:26:35,949
is 33%. So, and then there's this one. This is the policy adherence thing. And that's

314
00:26:35,949 --> 00:26:40,889
a very, very hard problem, but just knowing that the agent didn't do something creative,

315
00:26:41,609 --> 00:26:44,549
nefarious, whatever you wanna call it, to go out of bounds.

316
00:26:45,709 --> 00:26:50,689
And as a systems person kind of ramping up on this particular space, I learned there's

317
00:26:50,689 --> 00:26:54,969
like very interesting metrics that show up. Like at Amazon, we obsessed a lot about having

318
00:26:54,969 --> 00:26:59,329
the right metrics and spending a lot of time identifying the right metrics. And I found

319
00:26:59,329 --> 00:27:01,969
these two metrics to be very, very fascinating.

320
00:27:02,989 --> 00:27:07,429
One of them measures capability and the other one measures reliability. So there's pass

321
00:27:07,429 --> 00:27:13,629
at K and it means that if I run the task 10 times,

322
00:27:14,349 --> 00:27:19,689
at least one of those 10 times the agent will give me the right answer, okay? So now,

323
00:27:19,709 --> 00:27:24,869
why is this useful? Let's say I have an agent that costs 1 % of the frontier models,

324
00:27:25,349 --> 00:27:29,549
but I can really, really detect when I have the right answer. Then I can just run that

325
00:27:29,549 --> 00:27:34,089
agent 10 times and still come out ahead probabilistically, right? So that's where pass

326
00:27:34,089 --> 00:27:39,069
at K could be very, very useful because I can just hedge and get an answer.

327
00:27:40,509 --> 00:27:46,749
And this is like, can I do this at all, right? And there's gonna be like cheap agents

328
00:27:46,749 --> 00:27:51,329
that can, models that can just never do it. But if you can do it one out of 10 or whatever

329
00:27:51,329 --> 00:27:55,869
the K number is, that's a capability assessment.

330
00:27:57,809 --> 00:28:02,749
This one, like the weirdest thing, I was like trying to figure, I assume there's gonna

331
00:28:02,749 --> 00:28:07,729
be a very elegant name for this but this is literally pronounced pass hat K. And it means

332
00:28:07,729 --> 00:28:13,069
if I run it K times, am I gonna get the right answer each of the K times? And that is

333
00:28:13,069 --> 00:28:18,609
the reliability one. And this becomes more important when the outcome is harder to measure,

334
00:28:19,109 --> 00:28:22,949
right? When it's more subjective, it's more open -ended and you can't really detect,

335
00:28:22,989 --> 00:28:27,669
then you want like the reliable model to be doing your bidding.

336
00:28:30,669 --> 00:28:37,009
Like I would not put a, like one example of this is like Mark mentioned, like there are

337
00:28:37,009 --> 00:28:41,689
things that can like land you in jail if you like break policies, right? Like trade stocks

338
00:28:41,689 --> 00:28:46,389
doing, when you're not supposed to, you don't wanna be experimenting with, you know.

339
00:28:46,509 --> 00:28:50,549
If it gets it right one out of 10 times, like nine out of 10 times, you're in jail. And

340
00:28:50,549 --> 00:28:54,469
you also don't wanna run the experiment 10 times because then you might go to jail nine

341
00:28:54,469 --> 00:28:55,129
times.

342
00:28:56,329 --> 00:29:02,109
All right, so, when you're evaluating the system, you're running the experiments over

343
00:29:02,109 --> 00:29:03,929
and over and over again. There's repetition.

344
00:29:04,889 --> 00:29:09,949
And oftentimes we found ourselves where we would see a very high degree of variance on

345
00:29:09,949 --> 00:29:15,889
whether a agent or whether a model was able to complete the job. And it's very tempting

346
00:29:15,889 --> 00:29:21,909
to just roll up all of the results in an aggregate and assign a score. But as you start

347
00:29:21,909 --> 00:29:22,729
diving deeper,

348
00:29:24,069 --> 00:29:28,589
like we were initially very uncharitable to some of the models where, it scored very,

349
00:29:28,589 --> 00:29:33,229
very poorly. And it turned out, well, it was because we were just running the test and

350
00:29:33,229 --> 00:29:38,709
we started getting throttled on the models. And unless you looked into the traces to

351
00:29:38,709 --> 00:29:45,309
see what actually went wrong and addressed that kind of artifact inside of your

352
00:29:45,309 --> 00:29:47,629
evaluation harness, you would never know. This is why I was like, when the strands age

353
00:29:47,729 --> 00:29:55,069
But you have to attribute the variance. So if you get something that's one off, and understanding

354
00:29:55,069 --> 00:30:00,849
why it failed, sometimes it'll just give you a very easy black and white artifact. But

355
00:30:00,849 --> 00:30:05,529
other times it actually teaches you a little bit about how the models behave. And it

356
00:30:05,529 --> 00:30:10,429
gives you a little bit better mastery on how to steer them together. It builds a lot

357
00:30:10,429 --> 00:30:14,869
more mechanical empathy as you start diving into the variance between models and then

358
00:30:14,869 --> 00:30:21,809
within the models. And by the way, with the 429s and the 529 or 503s,

359
00:30:22,469 --> 00:30:28,289
you're very reliant on the model provider, right? So I know that again, cost gets talked

360
00:30:28,289 --> 00:30:34,989
about a lot. The model itself gets talked about a lot in sort of public forums.

361
00:30:35,309 --> 00:30:41,169
But the provider actually matters a lot. The same model on different providers will impact

362
00:30:41,169 --> 00:30:45,629
your performance significantly. And there's a diurnal pattern with almost all the providers

363
00:30:45,629 --> 00:30:49,669
nowadays just because of the shortage on GPUs. So if you run something during the day,

364
00:30:49,969 --> 00:30:55,329
it might be slower, it might give you more 429s, it might throttle you more. If you run

365
00:30:55,329 --> 00:31:02,609
it at night, the same tests may perform up to 2x, 3x better just because the fleet's

366
00:31:02,609 --> 00:31:06,689
lower utilization. And that sometimes is not even measurable because you don't have access

367
00:31:06,689 --> 00:31:11,269
to observability on their end. Again, this is why it comes back to repeatability. If

368
00:31:11,269 --> 00:31:15,029
you're repeating these tests automatically, you can observe some of the behavior and

369
00:31:15,029 --> 00:31:20,529
draw insights from it. We also learned, again, building this was a really good way to

370
00:31:20,529 --> 00:31:25,849
ramp up on this ecosystem as well. Some harnesses have a turn cap. You don't wanna go

371
00:31:25,849 --> 00:31:29,789
back and forth. As much as Clot Code will make it look like it just went and solved the

372
00:31:29,789 --> 00:31:35,709
problem on a one shot, sometimes you see dozens of turns as you start looking at it and

373
00:31:35,709 --> 00:31:40,909
you can start looking at the traces. It's fascinating. So one recommended thing I would

374
00:31:40,909 --> 00:31:47,449
say for everybody, just go read a prompt for one of the coding agents and then ask

375
00:31:47,449 --> 00:31:54,029
for why the Codex prompt talks about goblins. Sort of got it. At least until a

376
00:31:54,029 --> 00:31:58,649
month ago, it talked about do not under any circumstances talk about goblins. That was

377
00:31:58,649 --> 00:32:04,389
in the system prompt. Every time you typed anything into Codex, those sentences were

378
00:32:04,389 --> 00:32:09,069
being transmitted from your computer to the OpenAI web servers. Anybody have a guess

379
00:32:09,069 --> 00:32:11,229
as to why it's so specific?

380
00:32:14,629 --> 00:32:15,469
Okay, fair enough.

381
00:32:18,869 --> 00:32:23,509
It was part of the test criteria. One of the tests that were being used was trying to

382
00:32:23,509 --> 00:32:30,249
trick the agent into talking about goblins and go off path. And this is,

383
00:32:30,269 --> 00:32:37,469
by the way, a lot of, as you start reading these system prompts, you learn how

384
00:32:37,469 --> 00:32:42,269
seemingly primitive these conversations are. The quad code one was very much like,

385
00:32:43,849 --> 00:32:48,869
Important, very important. Very, very, very important. It's not very sophisticated. Literally,

386
00:32:49,069 --> 00:32:49,789
it's not joking.

387
00:32:50,709 --> 00:32:55,209
And it's all caps. It's like Michael Scott from The Office where he got tired of people

388
00:32:55,209 --> 00:32:59,389
not listening to his messages. So he started labeling everything as urgent and people

389
00:32:59,389 --> 00:33:03,989
stop paying attention to urgent. So he had urgent B and urgent C and urgent D. These

390
00:33:03,989 --> 00:33:08,449
prompts read like that. And if you wanna ramp up on how to commandeer your own agents,

391
00:33:08,449 --> 00:33:12,249
go read some of these system prompts. It's fascinating.

392
00:33:13,489 --> 00:33:20,209
But it also tells you about this, when it detects it's under evaluation, it behaves

393
00:33:20,209 --> 00:33:21,289
a little bit better.

394
00:33:22,709 --> 00:33:26,749
One of the things that we keep coming back to is the tracing and the observability of

395
00:33:26,749 --> 00:33:30,889
your harness. If you don't ask it to show the work, you don't really know what happened.

396
00:33:30,949 --> 00:33:36,109
And you don't know if you're actually measuring a system environment bug versus an actual

397
00:33:36,109 --> 00:33:39,569
logic bug or a shortcoming in your agent.

398
00:33:40,669 --> 00:33:44,589
One of the things that hasn't standardized here, unfortunately, is that each harness

399
00:33:44,589 --> 00:33:51,369
chooses on its own how to log and how to trace a given session,

400
00:33:51,469 --> 00:33:53,249
let's say, except for strands.

401
00:33:54,329 --> 00:33:58,749
Thankfully, it actually uses open telemetry in my previous life. I was working on open

402
00:33:58,749 --> 00:34:04,689
telemetry at an observability startup. So that was really great to see that they're kind

403
00:34:04,689 --> 00:34:11,249
of out of the gate supporting an open source standard model, standard format, so that

404
00:34:11,249 --> 00:34:15,989
it can be then ingested into the tool of your choice to understand it better. Unfortunately,

405
00:34:16,009 --> 00:34:21,269
for a lot of the other harnesses, you have to write your own parser and wrapper to measure

406
00:34:21,269 --> 00:34:27,469
and then to emit that data into your vendor of choice, depending on where you want to

407
00:34:27,469 --> 00:34:29,269
view and visualize the traces.

408
00:34:30,269 --> 00:34:30,669
So, yay.

409
00:34:32,949 --> 00:34:38,049
All right, we'll share a couple of more interesting results before we conclude. So, with

410
00:34:38,049 --> 00:34:41,429
all of that in mind, and it was important to share that preface before we share some

411
00:34:41,429 --> 00:34:44,909
of the results that we had. Ember One, what's Ember One?

412
00:34:45,109 --> 00:34:50,249
So, Ember One is this really cool custom model that Fireworks, who is an inference provider

413
00:34:50,249 --> 00:34:56,249
for open weight models, recently, I think last week, launched. It is a fine tune. It

414
00:34:56,249 --> 00:35:02,909
is a smaller, smaller -ish version of KimiK3, which is another very capable Trillian

415
00:35:02,909 --> 00:35:07,949
parameter open weight model on coding. And they did something that's really fun. They

416
00:35:07,949 --> 00:35:12,829
made the reasoning. Remember what Quadra said about higher reasoning effort, lower reasoning

417
00:35:12,829 --> 00:35:17,669
effort? They reduced the reasoning of KimiK3. They basically said, hey, you think too

418
00:35:17,669 --> 00:35:22,249
much. You're over -engineering everything. We're gonna train you on what type of productive

419
00:35:22,249 --> 00:35:27,269
thinking looks like versus what unproductive and spiraling looks like. And then they're

420
00:35:27,269 --> 00:35:31,349
gonna bake that into the model itself. So, release this. It's been working really well

421
00:35:31,349 --> 00:35:36,669
for us. Sort of, they're still in beta, so time will tell. But it's a really cool idea.

422
00:35:37,469 --> 00:35:42,169
And it's much faster and it reduces costs because a lot of the output tokens from reasoning

423
00:35:42,169 --> 00:35:46,669
can cost you a lot of money. Ever have a neighbor or that friend that just took a long

424
00:35:46,669 --> 00:35:53,469
time to get to the point? It's literally just, as far as I can tell, coaching that model

425
00:35:53,469 --> 00:35:58,669
to just get to the point faster. And as a result, it emits fewer tokens, so it costs

426
00:35:58,669 --> 00:35:58,929
less.

427
00:35:59,829 --> 00:36:04,149
And again, your output is just driven by the output velocity. That's the biggest barrier.

428
00:36:04,529 --> 00:36:08,569
So if you have to emit fewer things, fewer reasoning tokens, you get it. But it's very

429
00:36:08,569 --> 00:36:14,189
elegant. So anyways, we tried Strands with the Strands harness. And it's not meant to

430
00:36:14,189 --> 00:36:15,829
be a coding harness, I think.

431
00:36:16,929 --> 00:36:21,569
But the launch blog baited me into it because the launch blog talked about coding.

432
00:36:21,669 --> 00:36:24,689
But since the AWS folks baited me into it, we went and tried it. So on Ember, the same

433
00:36:24,689 --> 00:36:30,869
tasks, we came up with three very simple tasks on the Valkey code base, the open source

434
00:36:30,869 --> 00:36:32,869
code base. It's written in C.

435
00:36:35,069 --> 00:36:39,469
And it has a really good set of unit tests and things like that baked in. So the same

436
00:36:39,469 --> 00:36:45,869
task on, same three bugs on Ember one with Clot code were like $2 .34, whereas it was

437
00:36:45,869 --> 00:36:51,889
less than half the price with Strands. But why? Why is Strands cheaper?

438
00:36:54,209 --> 00:36:59,929
And it turns out, so you can look into a few things, like these metrics matter. How many

439
00:36:59,929 --> 00:37:04,749
turns is it taking? And how much context is it shipping and how much context is it getting

440
00:37:04,749 --> 00:37:11,309
back? Again, like who can remain more laser focused? And we found that

441
00:37:11,309 --> 00:37:18,289
Strands will often, like generally speaking, it takes smaller turns.

442
00:37:18,609 --> 00:37:25,169
Like it sends a lot of context back between itself and the model. And personally,

443
00:37:25,169 --> 00:37:31,149
I've found Clot code to be very verbose. Like even before Strands came out, what happened

444
00:37:31,149 --> 00:37:37,709
is in the world of KV caching, you want like the entire prefix has to match for you

445
00:37:37,709 --> 00:37:43,669
to be able to reuse the prompt and to not have to do the prefill again.

446
00:37:45,029 --> 00:37:49,269
And Clot code really optimizes for that. That's why you see that beautiful cache, like

447
00:37:49,269 --> 00:37:53,269
the 90 % off on cache. And if you look into your Clot code harness, you get a very high

448
00:37:53,269 --> 00:37:57,729
cache rate. If you were designing this as an engineer, what would you do if you wanted

449
00:37:57,729 --> 00:38:00,009
to optimize the cache rate between every turn?

450
00:38:03,759 --> 00:38:04,439
You just append.

451
00:38:05,519 --> 00:38:09,999
Every time somebody says something, you append the entire fricking context because you

452
00:38:09,999 --> 00:38:16,099
touch one word and the entire KV cache is invalidated. So now it becomes a self -fulfilling

453
00:38:16,099 --> 00:38:20,159
prophecy because they just have to, like to maintain a higher cache rate, they have to

454
00:38:20,159 --> 00:38:25,739
ship more tokens into it. And they keep doing this and it keeps getting longer and longer

455
00:38:25,739 --> 00:38:30,059
until you hit that compaction. And then you get like everybody is aware of what that

456
00:38:30,059 --> 00:38:33,259
feels like, right? When you have that compaction and it starts to like freeze you up.

457
00:38:34,439 --> 00:38:35,899
It's very verbose.

458
00:38:37,299 --> 00:38:42,359
Strands somehow is taking shorter turns. It's sending a lot less data back and forth.

459
00:38:43,099 --> 00:38:48,299
It's ruining your KV cache, but maybe it doesn't matter. And maybe it's not ruining the

460
00:38:48,299 --> 00:38:52,399
KV cache as much as you might think. So we'll get into that in a second. And then like

461
00:38:52,399 --> 00:38:56,879
another test, again, you want to test different things. We tried the Jin repository,

462
00:38:56,879 --> 00:38:59,799
which is a server in the Go world.

463
00:39:01,699 --> 00:39:04,479
And again, it was like a little bit closer.

464
00:39:06,199 --> 00:39:11,139
Where with Strands, same model, Opus 4 .8. And when we did this, the other option was

465
00:39:11,139 --> 00:39:17,159
Opus 5. And Opus 5 was performing much worse. So we stuck with 4 .8. That's the other

466
00:39:17,159 --> 00:39:19,259
thing. Don't ever assume that the newest model is the best.

467
00:39:20,399 --> 00:39:25,379
5 .5 is better than five in our experience, but 5 .0 sucks compared to 4 .8 in the test

468
00:39:25,379 --> 00:39:25,999
that we ran.

469
00:39:27,719 --> 00:39:30,999
But $3 .72 versus $4 .97,

470
00:39:32,259 --> 00:39:34,539
all tests were passing.

471
00:39:35,499 --> 00:39:40,059
And as we dug into it, like this is the aggregate, but there was one of the 10 tasks

472
00:39:40,059 --> 00:39:45,159
that Strands was more expensive on, but nine out of 10, it was cheaper. And overall,

473
00:39:45,599 --> 00:39:49,019
it was cheaper as well. So let's take a look.

474
00:39:50,619 --> 00:39:53,679
Again, comes down to more turns, third of the size.

475
00:39:54,879 --> 00:40:03,659
The height here, it tells you the amount of context, and each

476
00:40:03,659 --> 00:40:05,259
turn is a block.

477
00:40:06,219 --> 00:40:09,779
So it'll send more turns once in a while, which means there's more latency going back

478
00:40:09,779 --> 00:40:16,699
and forth. But it turns out that overall task latencies were actually about

479
00:40:16,699 --> 00:40:23,119
the same. So why we started digging into a K -clot code? Help me understand what's going

480
00:40:23,119 --> 00:40:25,299
on. Why is Strands so elegant here?

481
00:40:29,039 --> 00:40:35,019
And I started going down, my prediction was there's a major amount of context engineering

482
00:40:35,019 --> 00:40:38,799
that's happening. You don't wanna send the entire history

483
00:40:40,119 --> 00:40:45,639
to the LLM every time. But there's this really awesome optimization, and thank you for

484
00:40:45,639 --> 00:40:50,419
open sourcing it, because we can kind of see and take inspiration from this, is that

485
00:40:50,419 --> 00:40:56,979
every time the model requests a tool call, the harness makes a tool call, whatever,

486
00:40:57,719 --> 00:41:03,959
sometimes you're getting Git repo and you're publishing code inside that prompt. And

487
00:41:03,959 --> 00:41:08,479
what happens is over time, especially as your history gets longer and longer,

488
00:41:10,039 --> 00:41:15,859
that context gets longer and longer. And all you have is a bunch of lines of code that

489
00:41:15,859 --> 00:41:20,419
you referenced 20 minutes ago that have nothing to do with the task that you're working

490
00:41:20,419 --> 00:41:26,039
on, but K -clot code is sending that every time. What Strands is doing is, and it's all

491
00:41:26,039 --> 00:41:31,119
configurable, which was the other really nice thing. It just truncates tool context to

492
00:41:31,119 --> 00:41:36,539
a, each tool output is like by default to 1 ,000 characters, I think.

493
00:41:39,319 --> 00:41:43,919
And that means that if the model actually needs the code that was referenced there, it

494
00:41:43,919 --> 00:41:48,039
can ask for it. You can go get it again. But if it doesn't need it, don't bother the

495
00:41:48,039 --> 00:41:53,519
model sending it. And this is actually in terms of just like, it's a huge service to

496
00:41:53,519 --> 00:41:58,659
GPU capacity around the world to do this kind of stuff instead of just sending the entire

497
00:41:58,659 --> 00:41:59,079
thing.

498
00:42:00,939 --> 00:42:06,759
Also, the entire code base for the harness part is only 4 ,000 lines, as far as we could

499
00:42:06,759 --> 00:42:13,499
tell. And the system prompt does not talk about goblins. It's a very concise, inspectable

500
00:42:13,499 --> 00:42:17,259
systems prompt, which made me very, very happy.

501
00:42:17,359 --> 00:42:22,259
a lot of people, most developers I know prefer cloth code even over Codex. When you add

502
00:42:22,259 --> 00:42:26,819
features, guess what, your lines of code goes up. So there is a trade -off. I mean, a

503
00:42:26,819 --> 00:42:30,679
lot of it is the prompt engineering that Kwajale talked about. But a lot of it is like,

504
00:42:30,679 --> 00:42:34,859
oh, I like this feature. I like this other feature. Oh, it has this tool. And then it

505
00:42:34,859 --> 00:42:39,639
has to put all that in the system prompt. It has to, there's like 30 default tools or

506
00:42:39,639 --> 00:42:44,439
whatever. It has to tell the model about all the tools. The model has no idea. So that's

507
00:42:44,439 --> 00:42:48,559
the reason that cloth code also gets really bloated over time. Fun fact, you can actually

508
00:42:48,559 --> 00:42:53,079
tell cloth code how to do compaction and what to prioritize and what to throw away. And

509
00:42:53,079 --> 00:42:58,379
you can do the compaction on your own schedule as well. So you can tune this thing and

510
00:42:58,379 --> 00:43:01,339
make it perform better as well.

511
00:43:03,039 --> 00:43:09,799
All right, so we talked about this rigorous eval system. And we'll end with this.

512
00:43:10,679 --> 00:43:17,099
When you are testing a system, you're not just testing the efficacy of the model. And

513
00:43:17,099 --> 00:43:21,579
there's a Databricks paper from a couple of months ago that actually showed that in many

514
00:43:21,579 --> 00:43:27,899
cases, the harness actually matters more than the model itself. And, but you gotta zoom

515
00:43:27,899 --> 00:43:30,579
back out even further. It all starts with the human.

516
00:43:31,639 --> 00:43:35,579
And the human can write a very bad prompt or they can write a good prompt that's like

517
00:43:35,579 --> 00:43:39,639
well organized, well thought out. Sometimes I turn on the reasoning tokens and it's like

518
00:43:39,639 --> 00:43:43,519
literally judging me. Like this guy can't make up his mind. I've literally seen that

519
00:43:43,519 --> 00:43:49,259
in the reasoning tokens. But like a good prompter with a good prompt is where it starts

520
00:43:49,259 --> 00:43:54,979
with the right context. So you're not kind of circling around. And then the harness makes

521
00:43:54,979 --> 00:43:59,439
a big difference in terms of how it uses that context. It's trying to, you as a human

522
00:43:59,439 --> 00:44:04,119
are trying to express something. The prompt is trying to capture that intent. The harness

523
00:44:04,119 --> 00:44:10,679
is trying to synthesize that intent into a compiled thing that goes to the model, which

524
00:44:10,679 --> 00:44:15,639
then is deciding well, how much do I wanna think about this? And how much do I wanna,

525
00:44:16,319 --> 00:44:21,619
how hard do I wanna think about it and so forth. And we are not testing models. We learned

526
00:44:21,619 --> 00:44:25,599
very quickly that we weren't testing frontier labs or set the other labs. You're testing

527
00:44:25,599 --> 00:44:27,759
the entire end to end setup.

528
00:44:29,699 --> 00:44:34,459
Yeah, with that, we'll turn it over for any questions or and then yeah, you can try.

529
00:44:34,619 --> 00:44:39,779
We'll make the slides available online as well. So you can try it yourself. We've mostly

530
00:44:39,779 --> 00:44:45,519
captured everything that we use behind an MCP server that you can generate tasks, run

531
00:44:45,519 --> 00:44:46,059
them

532
00:44:47,139 --> 00:44:52,919
and look at the results. And run it in our sandbox. And we have like the 2E version of

533
00:44:52,919 --> 00:44:53,779
our slides too.

534
00:44:58,079 --> 00:45:04,739
Thank you. Hey there, great presentation. Thank you very much. So you said at the, you

535
00:45:04,739 --> 00:45:11,419
first generated the, I guess the eval questions, right? But when you're like generating

536
00:45:11,419 --> 00:45:17,739
with the LLMs, a lot of those questions are similar. So how did you make sure that you're

537
00:45:17,739 --> 00:45:23,799
actually covering good enough sample space rather than they just look the clones? So

538
00:45:23,799 --> 00:45:30,339
a couple of things. One is that we started by, remember the whole measurable versus

539
00:45:30,339 --> 00:45:35,559
representative? So we started like measurable is like, you just programmatically go to

540
00:45:35,559 --> 00:45:40,359
a bug that got fixed, you roll it back and you generate a prompt. Like that, like you

541
00:45:40,359 --> 00:45:45,219
can, I'm sure like many, most complex repositories have plenty of bugs that you can kind

542
00:45:45,219 --> 00:45:49,779
of programmatically create. Now, but when you do that, it's really hard to know which

543
00:45:49,779 --> 00:45:50,479
bug to pick.

544
00:45:51,739 --> 00:45:56,139
And that's where it's like, it's super measurable, but is it representative? And so you

545
00:45:56,139 --> 00:46:01,379
don't know. And that's where you say, okay, I'm going to employ an LLM to come up with

546
00:46:01,379 --> 00:46:05,699
the, like a way to capture the essence of this repository and the essence of these tasks.

547
00:46:05,759 --> 00:46:12,279
And that gets very subjective. Now, I haven't found the models to be repetitive or

548
00:46:12,279 --> 00:46:16,139
like they're not all producing the same thing, but I can tell you like what my workflow

549
00:46:16,139 --> 00:46:22,639
is. I went on the LLM task generation bits, we actually end up employing a variety of

550
00:46:22,639 --> 00:46:28,119
models to generate the task. We want to spend a lot of money, a lot of tokens on creating

551
00:46:28,119 --> 00:46:32,779
that corpus of tasks. And then we merge those, and then we have another LLM judge that's

552
00:46:32,779 --> 00:46:37,259
going through and saying, okay, which one of these are the best? Now that sounds really,

553
00:46:37,259 --> 00:46:43,339
really simple to state, but the level of entropy here is extremely high. To get it right,

554
00:46:43,359 --> 00:46:47,479
it's very hard. And that's the real tension between building measurable tasks

555
00:46:50,039 --> 00:46:54,499
and representative tasks. And the human input part of this is really important, especially

556
00:46:54,499 --> 00:46:59,959
for existing repos. If you have SME subject matter experts on the repos, you need to

557
00:46:59,959 --> 00:47:05,679
ask them. They also need to look at the output because part of it is like human computer

558
00:47:05,679 --> 00:47:11,079
collaboration. It's like, here's the set of tasks. Did I cover everything? Usually, like

559
00:47:11,079 --> 00:47:15,379
Kwajale said, if you utilize an LLM, not even the frontier model, you can use a smaller

560
00:47:15,379 --> 00:47:20,359
model. It's pretty good if you give it the history. It's pretty good at looking at your

561
00:47:20,359 --> 00:47:24,639
history and labeling your history. Like, oh, here's the infra setup. Here's where you

562
00:47:24,639 --> 00:47:28,839
did some optimizations. Here's where you fixed the bug. Here's a new feature. So it tries

563
00:47:28,839 --> 00:47:32,739
to categorize, and you need to have good representation across the categories. And if

564
00:47:32,739 --> 00:47:36,539
in thought. Yeah, so just quick follow up on that. So it sounds like you guys created

565
00:47:36,539 --> 00:47:42,799
a golden data set, got the SME inputs as well. And when you're employing the LLMS judge,

566
00:47:43,219 --> 00:47:48,179
did you guys use one, came up with the rubrics? And if so, how did you come up with rubrics

567
00:47:48,179 --> 00:47:50,259
as well? Because that's also very time consuming.

568
00:47:51,379 --> 00:47:56,199
This is where we were talking about where we have to eval our evals. Like, it's a very,

569
00:47:56,199 --> 00:48:00,899
very hard problem. And we started by saying, we're gonna go to every enterprise and we're

570
00:48:00,899 --> 00:48:07,239
gonna generate an eval golden set for all of their repositories. And then we realized

571
00:48:07,239 --> 00:48:11,519
that that is a very complex problem that we're not quite ready to solve yet. So what

572
00:48:11,519 --> 00:48:16,239
we have been trying to do is to just create a collection of problems that look similar.

573
00:48:16,679 --> 00:48:23,299
So that we can still get like approximation of how a particular model and

574
00:48:23,299 --> 00:48:27,219
harness setup is doing for tasks that look similar to them as opposed to just looking

575
00:48:27,219 --> 00:48:32,579
at sweet bench or the 50 other very generalized kind of tasks.

576
00:48:33,679 --> 00:48:40,239
Like that specificity really matters. Like having like

577
00:48:40,239 --> 00:48:46,819
domain specific capability and fine tuning can make a big difference on the

578
00:48:46,819 --> 00:48:47,179
performance.

579
00:48:47,379 --> 00:48:52,779
So I know one of the criteria for evaluation was the traces of the model. Are those like

580
00:48:52,779 --> 00:48:56,939
always easy to read? Is it always easy to tell? Or do you find maybe you're picking models

581
00:48:56,939 --> 00:49:01,339
where the traces are just maybe harder to read and you can't tell it was cheating? Oh

582
00:49:01,339 --> 00:49:05,899
yeah, this is why I was like, it's really, really important to try to pick a standard

583
00:49:05,899 --> 00:49:10,259
early. It's already kind of too late because there's already too many harnesses. But

584
00:49:10,259 --> 00:49:14,779
I think setting an example, like strands coming out with open telemetry and being like,

585
00:49:14,859 --> 00:49:17,759
okay, like why doesn't everybody just follow the same format?

586
00:49:18,739 --> 00:49:25,279
I think most harnesses other than strands, it's just JSON or not even JSON. Some of

587
00:49:25,279 --> 00:49:30,339
them are just streaming free form text, which does become expensive and time consuming

588
00:49:30,339 --> 00:49:35,399
to parse, especially when you're trying to visualize. What we ended up doing is we actually

589
00:49:35,399 --> 00:49:40,739
had to write our own schema to normalize the traces across different harnesses. And then

590
00:49:40,739 --> 00:49:44,979
we only had to write the visualizer once. So then we can visualize it the same way. But

591
00:49:44,979 --> 00:49:48,839
it's definitely not foolproof because each of the harnesses have their own ways of capturing

592
00:49:48,839 --> 00:49:55,539
tool calls, tool outputs, the reasoning tokens, and just like the different system environment

593
00:49:55,539 --> 00:50:00,099
messages that get included in cloud code, for instance. To be honest, it's kind of random.

594
00:50:00,379 --> 00:50:02,939
I feel like sometimes they add a feature and they're just like, oh, we're just gonna

595
00:50:02,939 --> 00:50:07,259
put this at the end. And you're like, oh, but you have another field that literally says

596
00:50:07,259 --> 00:50:11,439
system environment. Shouldn't it go in there? And it's like, okay, well, they're shipping

597
00:50:11,439 --> 00:50:14,699
fast and doing things that you can't predict.

598
00:50:15,779 --> 00:50:20,499
It's definitely observability I think is still a pretty big problem in harnesses especially.

599
00:50:22,099 --> 00:50:28,539
What does your experience tell you about harnesses? Because to me, it almost feels like

600
00:50:28,539 --> 00:50:34,199
you want to control your harness. And it's great having a coding harness, but that's

601
00:50:34,199 --> 00:50:38,899
just the beginning. The moment you want to do more interesting things, you're just gonna

602
00:50:38,899 --> 00:50:44,799
want to control and you cannot depend on the automatic updates that come with the default

603
00:50:44,799 --> 00:50:49,779
harnesses out there. We ran into a surprising number of customers that were like, I'm

604
00:50:49,779 --> 00:50:53,319
gonna build my own coding harness. And I started judging them at first, but then I realized

605
00:50:53,319 --> 00:50:55,079
Momento also built its own coding harness.

606
00:50:56,739 --> 00:51:03,379
We wanted to see how we can beat cloud code and I think we did in using

607
00:51:03,379 --> 00:51:09,439
similar kind of techniques. It's very tempting, but the ecosystem is moving so, so quickly

608
00:51:09,439 --> 00:51:13,819
and there's new techniques coming out. So if you are going to encode your own behavior,

609
00:51:13,999 --> 00:51:19,439
you also have to do this management exercise of constantly making sure that you're employing

610
00:51:19,439 --> 00:51:24,419
the latest techniques. Like this context engineering aspect or like tool call truncation.

611
00:51:24,879 --> 00:51:29,319
If you had like had cloud code, look at the cloud code code and generate your own harness

612
00:51:30,899 --> 00:51:33,319
and you hadn't seen this particular capability,

613
00:51:35,259 --> 00:51:39,359
you may miss out. That said, I think it's probably a lot easier to build your own coding

614
00:51:39,359 --> 00:51:43,679
harness just by using the agent harness as a standard agent harness as a starting point.

615
00:51:43,779 --> 00:51:48,219
Yeah, I think exactly. Like use a foundation, don't start from scratch. There's a ton

616
00:51:48,219 --> 00:51:52,539
of women's source ones. Strands obviously we've talked about a lot tonight. Pi is another

617
00:51:52,539 --> 00:51:53,159
good one.

618
00:51:54,119 --> 00:51:59,179
Yeah, with plugins, I absolutely think everybody should eventually get to the point where

619
00:51:59,179 --> 00:52:04,739
they customize their harness. Don't build it from scratch, but plugins is a great way,

620
00:52:04,819 --> 00:52:08,059
especially for enterprises, you always have like secret sauce, special sauce, special

621
00:52:08,059 --> 00:52:12,859
tools that general generic harnesses do not have access to. And you can just build that

622
00:52:12,859 --> 00:52:17,079
in as a plugin so that it's a default to everybody that works at a given company. But

623
00:52:17,079 --> 00:52:20,479
you're absolutely right. I feel like the harness is so important, sometimes more than

624
00:52:20,479 --> 00:52:27,139
the model that you should be in control of it. And you would never not control like

625
00:52:27,139 --> 00:52:28,579
what tools you give your engineers.

626
00:52:31,739 --> 00:52:37,019
Yeah, I mean, I really liked that point. And Strands harness should be a great place

627
00:52:37,019 --> 00:52:39,559
to build on as you think about building your own harness.

628
00:52:40,599 --> 00:52:46,399
That was one of our goals there. A question for you guys. You talked about sources of

629
00:52:46,399 --> 00:52:47,479
non -determinism.

630
00:52:48,539 --> 00:52:53,619
One of the conspiracy theories is that some inference providers more aggressively quantized

631
00:52:53,619 --> 00:52:56,999
during the peak of the day. Did you see any of that in your measurements?

632
00:52:58,959 --> 00:53:03,219
So there's an experiment artifact in a lot of my experiments. A lot of my experiments

633
00:53:03,219 --> 00:53:06,639
run at night while I, you know, for the minimal time that I sleep.

634
00:53:08,179 --> 00:53:12,039
And I was thinking about this as Daniella mentioned this diurnal pattern. I was like,

635
00:53:12,239 --> 00:53:17,219
man, like that's an experimental artifact that I haven't really zoomed in on because

636
00:53:17,219 --> 00:53:22,879
we just run our stuff at nights. And then we spend the day analyzing them. But that,

637
00:53:22,879 --> 00:53:27,899
so I don't have data on quantization. But there's, it's a more,

638
00:53:28,879 --> 00:53:34,059
it's a very hard to detect problem. It's like asking somebody, what is the ping latency

639
00:53:34,059 --> 00:53:36,079
between two EC2 instances in the same AZ? You just like, there's like, or it's like saying,

640
00:53:36,079 --> 00:53:36,899
hit. What are the odds that I don't get hit if I did it a thousand times? Because load

641
00:53:36,899 --> 00:53:43,419
imbalance can happen even in periods of low load. And you might

642
00:53:43,419 --> 00:53:46,959
still land on node that has, you know, that's over capacity.

643
00:53:48,399 --> 00:53:53,239
And there might be signals, like there could be bugs in the model provider, like a deployment

644
00:53:53,239 --> 00:53:59,179
could load the wrong way to the wrong time of day. So it's really, really hard to, for

645
00:53:59,179 --> 00:54:04,259
me to accurately assess if they were intentionally doing it. I'll say this though. I

646
00:54:04,259 --> 00:54:09,459
have an interesting anecdote. So I did want to test that out. I ran a test across three

647
00:54:09,459 --> 00:54:16,139
providers. It was Base 10, Fireworks and Together AI. And it was GLM 5 .2. It was

648
00:54:16,139 --> 00:54:22,019
before 5 .3 came out for 24 hours. So that I could see, and the variance between providers

649
00:54:22,019 --> 00:54:27,159
was smaller than the variance within a provider during the different times of day. So

650
00:54:27,159 --> 00:54:32,399
quantization, not sure, cause they might have to like reload the model, but they definitely

651
00:54:32,399 --> 00:54:37,879
switched their fleets around a lot. And the variance was huge, right? Like you would

652
00:54:37,879 --> 00:54:41,779
imagine that between two providers, you would see more variance, but I think they all

653
00:54:41,779 --> 00:54:45,599
use the same GPU optimization techniques behind the scenes

654
00:54:46,539 --> 00:54:52,859
because it just looked like night and day, which again, like it made some of the reproducible

655
00:54:52,859 --> 00:54:58,139
results really annoying because we just had to repeat like five, five, six times in order

656
00:54:58,139 --> 00:55:03,999
to feel confident in a result. I do think that providers should offer the quantization

657
00:55:03,999 --> 00:55:06,639
level as part of the model name.

658
00:55:08,579 --> 00:55:11,919
So yeah, it's a lot of effort behind the slides.

659
00:55:13,359 --> 00:55:18,679
Here, because I'm in a startup, right? I'm not the like big corporation. So I have kind

660
00:55:18,679 --> 00:55:23,819
of a little bit different of views than what you guys trying to achieve here because

661
00:55:23,819 --> 00:55:29,039
as a startup, now number one goal is to create a product, right? Ship it, right?

662
00:55:29,939 --> 00:55:34,979
So I cannot do all these kind of, you know, evil for the code agents.

663
00:55:36,479 --> 00:55:42,999
And when you're building a startup, I feel like all this coding agent for me is not an

664
00:55:42,999 --> 00:55:48,379
entry -level summer intern. Write some code, give them a test and give me a reading and

665
00:55:48,379 --> 00:55:48,719
greeting.

666
00:55:49,639 --> 00:55:53,319
I view them, each of these agents are like principle data managers or something like

667
00:55:53,319 --> 00:55:58,239
that behind. You know, kind of work for me, right? So when you can evaluate a principle

668
00:55:58,239 --> 00:56:04,739
engineer or whatever, I don't think we should pick the evil or give them

669
00:56:04,739 --> 00:56:07,399
a test, give them a greeting. We put them to compete,

670
00:56:08,459 --> 00:56:13,839
right? So I think this is kind of fundamental philosophy different than you do here.

671
00:56:14,319 --> 00:56:20,279
I feel like, you know, if I, as a startup guys, I will not using evil this way to kind

672
00:56:20,279 --> 00:56:26,319
of evaluate them. I will put them to compete with each other. I don't care who can get

673
00:56:26,319 --> 00:56:32,259
it done better, get cheaper or whatever, get it done, right? And I based on result to

674
00:56:32,259 --> 00:56:35,119
manage to pick which one will do more. For example, based on my experience, I use cloud

675
00:56:35,119 --> 00:56:40,179
innovating AI age. So I think you're absolutely right. If you make agents compete, the

676
00:56:40,179 --> 00:56:45,439
method by which you will tell which one is better is the evals bit. And you will never

677
00:56:45,439 --> 00:56:50,739
know who might have done it better or worse unless you run all of them. But if you run

678
00:56:50,739 --> 00:56:56,679
multiple, then you have to, you know, kind of assess their completion. That said, the

679
00:56:56,679 --> 00:57:00,599
big takeaway from this is not that I'm not telling everybody to go run their eval system.

680
00:57:00,819 --> 00:57:04,259
My big takeaway that I hope to convey is that be skeptical.

681
00:57:05,399 --> 00:57:10,759
What works as the best thing for a particular solution today may not be the best solution

682
00:57:10,759 --> 00:57:16,459
for that particular problem. And you see this, like if you look at open routers, traffic

683
00:57:16,459 --> 00:57:18,499
as well, like you'll see how traffic shifts

684
00:57:19,779 --> 00:57:24,379
between, you know, entropic and open AI models over time because people are just discovering

685
00:57:24,379 --> 00:57:28,999
that they're different. So remain skeptical, disconfirm your beliefs. Today, Codex might

686
00:57:28,999 --> 00:57:32,539
be really good at a particular task with Astra and tomorrow it might be the worst of

687
00:57:32,539 --> 00:57:38,859
the batch. As long as you can continue evolving your setup, it's probably fine. I think

688
00:57:38,859 --> 00:57:43,399
here it's more like one agent review others. For example, right? What I do is ask a cloud

689
00:57:43,399 --> 00:57:48,379
code writer spec. But now it's Codex to kind of review the spec and give me all the kind

690
00:57:48,379 --> 00:57:53,219
of hard feedback. And later I will do is maybe cloud, you can say incorporate all the

691
00:57:53,219 --> 00:57:58,739
feedback from Codex then kind of get a new version of the spec, right? So then kind of

692
00:57:58,739 --> 00:58:03,079
multiple version of this kind of work then you kind of, you will read the result at that

693
00:58:03,079 --> 00:58:07,499
time. You don't want to read at the very beginning, right? So then you have kind of decent

694
00:58:07,499 --> 00:58:12,619
spec to start to build something new. So that's kind of my experience. Using agents and

695
00:58:12,619 --> 00:58:16,799
adversarial capacity against each other is a really good idea. My workflow has that like

696
00:58:16,799 --> 00:58:20,319
automated as well. Like, so I think like using one to code review and having both of

697
00:58:20,319 --> 00:58:23,279
them code review. I think the most profound point of what you're saying

698
00:58:24,479 --> 00:58:29,079
is sometimes it's the output quality and like being able to one shot something is much

699
00:58:29,079 --> 00:58:35,559
more important than the dollars spent on it because your time matters a lot and that

700
00:58:35,559 --> 00:58:39,339
changes how much you can focus as well. We'll take one last question and then I think

701
00:58:39,339 --> 00:58:40,859
we'll be around after, but.

702
00:58:42,949 --> 00:58:45,489
Oh, sorry. I had one. Or we can take both.

703
00:58:47,909 --> 00:58:51,589
I was just curious, what's the, is you guys did all the different things you came up

704
00:58:51,589 --> 00:58:56,469
with. It was like the longest or hardest one. Like, were you, did you have any of your

705
00:58:56,469 --> 00:59:01,169
things that ran into the hours, multiple hours, multiple days that you'd have? Yeah,

706
00:59:01,169 --> 00:59:07,169
what's kind of the long end? Yeah, right now we're like, we're in the very, very early

707
00:59:07,169 --> 00:59:11,629
innings. So all our stuff is short horizon and one of the things that I called out is

708
00:59:11,629 --> 00:59:15,569
like a lot of the tasks that we have are not representative of a long horizon task, like

709
00:59:15,569 --> 00:59:19,509
a very diluted context history and so forth. So we haven't quite gotten that far. But

710
00:59:19,509 --> 00:59:23,049
like numbers, so like everything's right now under an hour. I take it the tasks that

711
00:59:23,049 --> 00:59:27,309
you're doing. Yeah, a couple hundred turns. No, no, no, it's about like, I think with

712
00:59:27,309 --> 00:59:32,289
grading and everything, it's around an hour because there's a suite, right? We do run

713
00:59:32,289 --> 00:59:37,629
public benchmarks. So like suite bench and terminal bench take multiple hours. Well,

714
00:59:37,769 --> 00:59:43,009
not the whole benchmark, the individual tasks. Yeah, so both suite bench and terminal

715
00:59:43,009 --> 00:59:49,069
bench have long horizon tasks that take multiple hours. We haven't come up with our own

716
00:59:49,069 --> 00:59:55,449
that take multiple hours, yeah. But it's one of the things that I think even in the research

717
00:59:55,449 --> 00:59:59,949
community is being talked about a lot because if you think about long horizon tasks in

718
00:59:59,949 --> 01:00:06,009
real life, rarely do they never require human input. But that's what these long horizon

719
01:00:06,009 --> 01:00:11,209
tasks are is that they're isolated and they require zero human input. I've had a few.

720
01:00:15,409 --> 01:00:15,669
Yep.

721
01:00:19,449 --> 01:00:25,949
This thing is, it's evaluating the agent itself, not the model, right? Meaning like,

722
01:00:25,969 --> 01:00:32,489
for example, when you use co -pilot and it routes different kinds of

723
01:00:32,489 --> 01:00:39,269
tasks to different kinds of models, then it can get the cheap tasks to the cheap models

724
01:00:39,269 --> 01:00:44,549
and the more expensive tasks and the more expensive models. Is that correct? We haven't

725
01:00:44,549 --> 01:00:48,889
gotten quite that sophisticated. We've built our own routing infrastructure and so forth,

726
01:00:48,949 --> 01:00:53,689
but we haven't tested. There's a whole lot of work to be done here. Testing what happens

727
01:00:53,689 --> 01:00:57,749
to different skills, what happens to different routing philosophies. There's a whole

728
01:00:57,749 --> 01:01:03,089
lot. We're just kind of getting started. But here, we did a very controlled set of experiments

729
01:01:03,089 --> 01:01:08,929
on one -shotting or surveying some long horizon tasks against a specific model with a

730
01:01:08,929 --> 01:01:11,889
specific harness at a specific reasoning level.

731
01:01:13,009 --> 01:01:18,689
All right, and then one thing I was really interested in is have you had much success

732
01:01:18,689 --> 01:01:24,689
testing things that are more fuzzy, like readability? For example, if you have an agent

733
01:01:24,689 --> 01:01:30,969
write one function that's 500 lines, that's probably bad. But also, if you have it write

734
01:01:30,969 --> 01:01:34,209
50 functions that are five lines, that's also probably really bad.

735
01:01:35,669 --> 01:01:41,289
That goes into the representative versus measurable part as well, because readability

736
01:01:41,289 --> 01:01:42,629
is also very, very subjective.

737
01:01:43,729 --> 01:01:48,649
You often have, as I mentioned, that one engineer that has very specific kind of taste.

738
01:01:49,429 --> 01:01:54,769
And every organization has these different standards. Now, you can run linter rules and

739
01:01:54,769 --> 01:01:58,949
so forth. That's very, very measurable, but that's where you start getting into LLM as

740
01:01:58,949 --> 01:02:05,809
a judge. To be able to assess how well this does compared to the essence of the Git repository

741
01:02:05,809 --> 01:02:10,889
as well. In the evaluation framework that we have right now, we do include an LLM judge

742
01:02:10,889 --> 01:02:16,249
that looks at conventions and readability and maintainability. A lot of that is just

743
01:02:16,249 --> 01:02:21,009
regular coding standards. What we really wanna be able to do is actually scan your repo

744
01:02:21,009 --> 01:02:25,689
to see what you prefer as an organization. There's also other things like risk tolerance,

745
01:02:26,109 --> 01:02:32,769
right? At certain organizations, you tolerate less code that can sort of

746
01:02:32,769 --> 01:02:38,009
fail in unexpected ways, like throw an error. That's an unclot exception, things like

747
01:02:38,009 --> 01:02:43,889
that. And if you've ever used this skill inside Clot Code, Clot Convention Review, it's

748
01:02:43,889 --> 01:02:48,789
very similar. It's basically like a convention review judge that you apply in addition

749
01:02:48,789 --> 01:02:52,929
to the test passing, which is correctness, you have the subjective test that's basically

750
01:02:52,929 --> 01:02:57,729
the LLM that's not like a random LLM. You have to prompt the LLM to check for specific

751
01:02:57,729 --> 01:03:02,429
things. Thank you for having us. And thank you Antithesis and Chris for sponsoring.

752
01:03:04,129 --> 01:03:05,169
Thank you.

753
01:03:06,329 --> 01:03:06,369
Yeah, I have one quick question, Mark, for you. Someone asked, your slide said