The full recordings and clips from the September 30 meetup at SURF Incubator: the opening, Marc Brooker on temporal policy for AI agents (Dogwood), and Khawaja Shams & Daniela Miao on evals for coding agents.
Deep engineering talks, right here in Seattle: run by engineers for engineers, company-neutral, no sales decks, monthly and free. Marc, Khawaja, and Daniela show what the talks are like, and Sitesh does, like, 85% of the work. Find the next one at seattlesystems.xyz. Music: “Voxel Revolution” by Kevin MacLeod, CC BY 4.0.
Seattle Systems is a meetup for people who build the hard parts: databases, distributed systems, and lately the agents that sit on top of them. The format is simple. Two deep technical talks, a room that fills up, and food you did not have to pay for.
This night was exactly what these nights are. Marc Brooker walked through a new language for governing what agents are allowed to do, with real math and real database tradeoffs. Khawaja Shams and Daniela Miao showed what it takes to evaluate coding agents, stumbles first, because speakers here bring their stumbles rather than their launch decks. The questions from the room were the kind you only hear from people who have been paged for the same problems. The conversations outlasted the recordings.
“Sitesh does, like, 85% of the work.”
From the opening remarks, and accurate. Seattle Systems wouldn’t be Seattle Systems without him.
If you build systems in Seattle, come to the next one. There will be food, courtesy of the Antithesis folks. Do not trust them.
Sitesh’s welcome, the Seattle Systems story, sponsor & speaker intros.
00:20 Dogwood & temporal policy · 02:36 Why deterministic policy · 06:05 Policy examples · 12:27 Why not Python/Rust/TS · 17:19 Open source runtime · 19:37 Compiling to SQL · 21:56 Concurrency control · 28:12 Temporal logic foundations · 33:47 Q&A
00:00 Opening & systems framing · 02:08 What an eval is · 06:15 Reasoning-effort surprises · 12:29 Your repo has culture · 19:05 Agents find the answer key · 28:29 pass@k: capability vs reliability · 42:43 Harness lessons · 45:30 Testing the whole setup · 53:50 Q&A
Sets Dogwood apart from LLM-as-judge/heuristic guardrails and explains why systems engineers should care.
The whole agent-governance problem in one clean, high-share clip: useful autonomy versus hard boundaries.
Concrete security example that stands alone: reading confidential data and sending email are safe separately, risky in sequence.
Highly relatable dev-agent policy: no commit unless the immediately preceding relevant action was a successful test.
A crisp argument for DSLs: the subtlety exists either way, but Dogwood makes it more visible and analyzable.
Punchy formal-methods payoff: prove a million-line policy will never spend over $1,000.
Great database/systems hook: temporal policy maps to SQL, and strong consistency becomes a policy requirement.
Turns database isolation into a vivid policy correctness problem: one approval, two concurrent sales.
Nice explainer segment: temporal operators make before/after/since policies mathematically grounded.
Useful testing/process clip: write failing cases and maybe generate them with property-guided fuzzers.
Strong Q&A answer for broad-capability agents: deny listing fails, allow listing is the safer mental model.
Clear boundary-setting clip: policy needs isolation and interception; Dogwood is part of the box, not the whole box.
Khawaja and Daniela trade the best lines: token costs, repo culture, agents that cheat creatively and optimize the checklist, the all-nighter the harness invalidated, and the takeaway: remain skeptical, disconfirm your beliefs. Music: “Voxel Revolution” by Kevin MacLeod, CC BY 4.0.
Makes the case for continuous coding-agent evals because models, harnesses, and repos keep changing.
Strong counterintuitive hook: more reasoning is not just slower and pricier, it can be less accurate.
Great framing for enterprise evals: PRs, rejections, tests, and history encode what generic benchmarks miss.
Names a hidden eval failure mode: benchmark prompts can become unrealistically helpful and unrepresentative.
Funny, concrete war story: agents inspect Git history, local checkouts, and neighboring processes unless the harness blocks them.
Explains eval-awareness without anthropomorphizing: agents optimize visible grading criteria like humans optimize checklists.
Clear systems-friendly explanation of pass@k and why cheap models can still be useful if success is detectable.
Memorable contrast between capability and reliability, with a punchy high-stakes example.
Turns noisy eval runs into an engineering practice: inspect traces, explain variance, build mechanical empathy.
Deeply practical harness lesson: tool output truncation can cut context bloat without losing recoverability.
The thesis of the talk in one clean arc: evals are not model benchmarks, they are system measurements.
Good Q&A clip for practitioners: let agents compete, but keep re-testing because today's best setup can decay tomorrow.