Harness over model
Teams are getting more out of improving the harness around a model than out of waiting for the next model.
-
Harness engineering is moving in-house
Up from the hobbyist end of this thread filed August 25. Gergely Orosz, August 25: Ramp built its own agent, Inspect, because engineers “wanted to run more than a few agents in parallel”; it now authors 75% of merged PRs. Two papers submitted August 24 and 25 freeze the base model, evolve only external memory and verification, and report +10 points on Terminal-Bench 2.0. Orosz
-
Harness investment now pays because the model curve stopped absorbing sloppiness
This moves the harness thread filed Tuesday from who builds one to why anyone should. Drew Breunig, quoted by Simon Willison on August 23: “Prior to Fable, it felt silly to waste too much time improving your coding harness or context strategies. A new model would arrive at the same price (or cheaper!) and paper over most of your problems.” Willison
-
The bottleneck moved from writing code to knowing what the agents wrote
This moves the Ramp harness thread filed Wednesday from who builds the agent to who reads its output. Thorsten Ball, August 30, argues for "the spot right in the middle, where speed and defects are in balance," shortening bug lifetime instead of chasing zero. Gergely Orosz, August 25: Inspect now authors 75% of Ramp's merged PRs, reported as a win. Both argue; neither measures review capacity. Ball · Orosz
-
New agent tooling is shipping harness-agnostic by default, not built around one vendor's CLI
Hamel Husain's evals-skills shipped Codex plugin packaging August 31, keeping installation "agent-neutral." Shreya Shankar's new DocWriter, built through August with CMU and UC Berkeley, connects Claude, Codex, Cursor or Pi as the drafting agent. DeepSeek Harness v0.1.2-alpha (August 27 and 30) runs Claude Code and Codex as installable subagents inside its own orchestration. All three demonstrated, not announced. Husain · Shankar · DeepSeek Harness
-
SHIPMeta graduates Muse Code out of beta with an SDK and subscriptions
August 31: inter-session messaging, workflow orchestration for parallel subagents, and a TypeScript SDK in developer preview that talks to a local muse host with no server or network involved. Three subscription tiers now undercut pay-as-you-go Meta Model API pricing. Meta
-
Code review is moving from a human gate to something the harness does for itself
Richard MacManus at Latent Space, September 1: Vercel's AI SDK, Astro and tldraw now replace external pull requests with agent "software factories" that triage, fix and review before a human merges — Vercel's factory authors 25 to 35% of its own merged PRs. Gergely Orosz's five-year retrospective the same day names code review "going optional" as the shift he expects through 2026. A Show HN project found one vocabulary cluster in Claude-authored PRs grew from 0.7% to 39% of a 461,000-PR corpus between early 2025 and mid-2026. MacManus · Orosz · Abraham
-
The gap in AI evaluation is a neglected discipline, not missing tooling
Hamel Husain's September 3 post extends his September 2 finding that automated judges catch most failures but miss the rest: he names five recurring pitfalls, from unvalidated LLM judges to synthetic test sets ungrounded in production data, and argues teams skip "looking at the data," the highest-ROI step, because it doesn't scale the way a tool does. Husain
-
Developers building coding-agent tools keep converging on the same designs because the same models are shaping all of them
Armin Ronacher's September 5 essay Latent Powers argues large models are quietly steering what developers choose to build, pointing to multiple people independently landing on similar projects. His own agent-tooling project, pi, shipped GPT-6 Astra support the day after OpenAI's launch — a demonstration of how fast that convergence moves, not just an argument for it. Ronacher · pi releases
-
Thorsten Ball argues engineers scrutinizing AI-written code line by line may be acting from habit, not from finding real defects. Ball
-
Three independent voices this week say line-by-line pull-request review no longer matches how code actually gets written
swyx's September 1 report found Vercel's AI SDK merges 25 to 35 percent of PRs from its own agent factory, while Astro, tldraw and Flue auto-triage or auto-close external PRs. Thorsten Ball, quoting Rachel Laycock, warns dropping review loses mentoring and shared ownership. Dex Horthy says his team's agent-only 'lights-off' pipeline rotted a codebase in three to six months — agents are rewarded for passing tests, not preserving design. swyx · Ball
-
An HN benchmark of 10 model/harness pairs found Qwen3.8-27B plus OpenCode fastest; a Claude variant sharpest but five times slower. Hacker News
-
Specialized testing techniques agents are told to use — TDD, formal verification, property-based testing, fuzzing — mostly perform worse than no instruction at all
Dan Luu's September 8 study ran 26 testing conditions, about 80 runs each, on Rust reimplementations of Zstd using GPT-5.6 Sol coding agents. Telling agents to do TDD produced more tests of worse quality; formal-methods agents proved properties irrelevant to the actual bug; fuzzing succeeded in roughly 5 of 160 runs. A hand-tuned custom skill did outperform the default, showing specific guidance can work where generic instruction can't. Dan Luu
-
Gergely Orosz: GitHub PR volume is up 5x in three years; risk-based review lifted Duckbill's merge rate 94%. Orosz
-
Code review is being redesigned around AI output, not just overwhelmed by its volume
Gergely Orosz's September 9 interview with OpenAI Codex lead Tibo Sottiaux moves past his own prior PR-volume numbers: Codex was deliberately built in Rust for long-term security despite weaker model support, and review is shifting to pre-code conversations about intent, agents handling correctness checks after. Sottiaux: architecture changes that 'used to take years can now take days.' Orosz · Orosz on PR volume
-
Armin Ronacher is now committing code to Mario Zechner's agent tooling repo pi-mono, not just writing about it. pi-mono
-
A single-source complaint about Astra's code quality now has a full technical case behind it
Armin Ronacher's September 7 post argues Astra's code is aggressively optimized for token efficiency, not human readability, and degrades without close oversight during long unsupervised runs — the reasoning behind an X post filed here September 9, where he called reverting to GPT-5.6 for daily work 'the first time I feel like this is a genuine regression.' Ronacher
-
Hacker News compared nine coding harnesses on a laptop: llama.cpp answered instantly, opencode took 20 minutes. Hacker News
-
Boris Cherny, via Willison: production Claude code needs a higher bar than human-written code. Willison
-
Telling a coding agent to follow standard testing methods often makes it worse, not better
Dan Luu's September 7 test of 26 prompting conditions on a real Rust Zstd implementation found agents given no testing instructions scored above average, while explicit TDD produced twice as many tests that enforced incorrect behavior and formal-verification tools like Verus and Lean 4 mostly proved properties unrelated to the actual bugs. Property-based testing was the one structured technique that helped. Luu
-
Claude Code v2.1.269 added claude plugin eval, scoring a plugin's eval suite with a reproducible JSON and HTML report. Claude Code changelog
-
Coding agents are starting to manage their own testing and coordination, not just write code
Thorsten Ball's September 12 newsletter describes routinely spawning isolated agent 'orbs' with Fable 5.1 and GPT-6 Astra that test features independently, generate video proof, and report back — with the lead agent adjusting its own prompts based on what they find, unsupervised. He calls it working at 'a higher altitude' than manual review allows. Ball
-
Agent transparency is becoming the harder problem than agent capability
Simon Willison's September 12 test had GPT-6 Astra chain OpenStreetMap tools into a working running route in 27 minutes — but the code behind it vanished once the conversation compacted, leaving no way to inspect how it was built. His point: agent frameworks need to preserve pre-compaction reasoning, not just final outputs. Willison
-
Jesse Vincent's episodic-memory v1.6.0 added a fifth supported harness and AWS Bedrock summarization, September 8. obra/episodic-memory
-
Claude Code v2.1.267 fixed a path-traversal bypass in marketplace plugin installs, September 9. Claude Code changelog
-
Harness engineering is consolidating around shared open code, not just competing essays
Armin Ronacher and Mario Zechner, previously known for separate writing on agent tooling friction, both shipped commits this week to Earendil's open-source 'pi' coding-agent harness — Zechner documenting its task-execution subsystem September 9 to 11, Ronacher cleaning up Codex WebSocket handling September 14. Demonstrated, not argued. Zechner commits · Ronacher commits
-
Agentic coding has outgrown the CI and test infrastructure built for human-paced commits
Anthropic's engineering team disclosed Sept 14 that Claude now authors roughly 80% of its own merged production code, with CI job volume up 25x in six months; Gergely Orosz reported Sept 15 that OpenAI's non-engineering teams went from near-zero Codex use to 90% adoption in four months, forcing continuous CI rework. Demonstrated, not argued. Anthropic · Orosz
-
A cheaper, narrower model class is emerging alongside frontier LLMs for structured decisions, not chat
Thorsten Ball's Sept 20 newsletter described TypeSafe's Jev as a non-LLM 'frontier-intelligence function call' — unstructured input, typed probabilistic decisions out, around 200ms latency — for tasks like shell autocomplete, code navigation and model routing. swyx's Latent Space separately tracked six independent open clones of Jev appearing within two days of release. Demonstrated adoption; Ball's read that it shifts engineering work toward product design is his own prediction. Ball
-
SHIPClaude Code ships Projects, a coordinator agent for parallel cloud sessions
The Sept 17 beta lets one coordinator agent direct parallel cloud-session threads with shared memory and artifacts. Creator Boris Cherny says it's now how he writes 'a lot' of his own code. MarkTechPost · Unite.AI
-
Practitioners agree agents have taken over routine coding execution, but split on what's left for human judgment
Thorsten Ball's Sept 20 newsletter quoted AWS's Marc Brooker saying 'humans have no role in routinely reviewing code' inside agent-driven pipelines. Gergely Orosz's Sept 17 interview with Matt Pocock argued the opposite emphasis: as agents write more code, engineering fundamentals — architecture judgment, taste — matter more, not less. Both are practitioner opinions responding to the same shift, not measurements. Ball · Orosz
-
ACTClaude Code proxy setups broke, then got fixed, within a day
Version 2.1.275 broke every request through a custom ANTHROPIC_BASE_URL proxy with a 400 error; 2.1.276 fixed it hours later. 2.1.273 separately reverted a permission-check fix, reviving prompts on previously approved Bash commands. Claude Code releases
-
Non-LLM decision models are emerging as a fast-copying product category running alongside chatbots, not replacing them
TypeSafe AI left stealth Sept 15 with Jev, a 'System One' model that outputs probabilities and confidence scores instead of text, built for judgments inside software rather than conversation. Latent Space documented six independent open clones within 48 hours; Thorsten Ball and Simon Willison each covered it independently Sept 20-21. Demonstrated adoption, not just a launch. TypeSafe AI · Latent Space · Willison
-
The debate over whether agents leave anything for human review is hardening into starker positions, not converging
Thorsten Ball's Sept 19 post argues code review is already effectively dead - humans can't audit agent-written code line by line in reasonable time - and predicts unit tests may follow. Gergely Orosz's Sept 17 counter, that engineering fundamentals matter more as agents write more code, stands unanswered. Ball's prediction, not a measurement; last week's split has widened. Ball · Orosz
-
The 'decision model' pattern is moving from announcement and clones to real practitioner tooling
Simon Willison shipped llm-typesafe, a working CLI plugin adding TypeSafe's Jev decision model to his llm tool, Sept 22 — a week after Latent Space counted six independent clones. Willison also endorsed 'decision models' as the better term over 'System One,' and Thorsten Ball separately praised Jev's approach Sept 20. Demonstrated shift from hype to shipped tooling, not just more clones. Willison · llm-typesafe release
-
The code-review debate is producing concrete alternatives now, not just more opinions
Design engineer Maggie Appleton told Gergely Orosz Sept 23 she's stopped reading AI-generated PRs, writing specs instead and coining 'capability gaslighting' for models that impress once then fail identically the next day. Separately, Claude Code creator Boris Cherny said this week he used Opus 5.5 with Lean and TLA+ formal verification on the Claude Agent SDK, yielding 16 bug-fixing pull requests from a few prompts. Two substitutes for review, not consensus. Orosz on Appleton · Cherny
-
MCP isn't being abandoned as agent tooling matures - it's narrowing to the jobs that actually need it
Simon Willison's Sept 20 post pushes back on the 'MCP was always a bad idea' argument circulating on Hacker News, saying MCP still matters for permission-scoped agent systems even as fully autonomous coding agents increasingly skip it for direct tool access. His own view, responding to a single thread; not yet corroborated elsewhere. Willison
-
The biggest lever on developer productivity has shifted from frameworks to how legible code is to an agent, not a human
Thorsten Ball argued Sept 26 that agents matter roughly a hundred times more than framework choice now, citing a roughly 100x drop in token cost over the past year - his own argument, not a measurement. Simon Willison dissented Sept 24, writing that the more he works with coding agents, the more convinced he is they make software engineering harder, not easier. Ball · Willison
-
A single success story with formal-verification tools doesn't hold up under controlled testing
Dan Luu's Sept 26 controlled test of 26 testing approaches on AI agents building a Zstd-like compressor found the default, no-special-instructions condition beat most guided techniques; formal methods including Lean and TLA+ clustered near or below average. That complicates Boris Cherny's Sept 22 account of 16 bug-fixing pull requests from a few Lean/TLA+ prompts. Demonstrated study versus one anecdote. Dan Luu · Cherny, via X
-
Harness engineering is optimizing what a plan needs to remember, not how much it writes down
Jesse Vincent's Superpowers 6.4.2, released Sept 25, cut planning time to a quarter and token use to about a third by recording only critical decisions - test assertions, signatures, spec values - instead of full code, while matching prior success rates on Sonnet 5 probes. A demonstrated engineering result. Jesse Vincent
-
Gergely Orosz reported DHH's 'death of coding by hand' call, broadening the agents-versus-craft debate, Sept 24. Orosz
-
DHH says 37signals wrote half as much code by hand in 20 months as in the prior 21 years. Orosz
-
Language and framework choice matters less than how well an agent performs in it
Thorsten Ball argued, recirculated via his Register Spill newsletter Sept 26, that agents are now 'a hundred times bigger' a productivity lever than any framework or language preference, urging developers to choose tools by what an agent handles well, not personal taste. His own argument, continuing the harness-over-model direction Geoffrey Huntley and Simon Willison have tracked this month. Ball
-
Context management is starting to move from a hand-written harness into a trained model skill
Nathan Lambert co-authored a University of Washington and Meta paper, posted Sept 29, that lets a model edit its own context as a file rather than follow a fixed compaction script. Trained this way, it beat hand-engineered context-management baselines by 11.4% on the BrowseComp-Plus benchmark using 21.5% fewer FLOPs. Demonstrated in a paper; not yet shipped in a product. arXiv · Lambert
-
Coding harnesses are starting to ship their own built-in supervisory agents, not just task-executing ones
Anthropic's Claude Code shipped a 'Mods' plugin system Oct 1, including a built-in 'You should know' side-agent that watches the main agent's work and flags what it might miss. The same day, Mario Zechner's independent 'pi' harness hit v1.0.0 with a durable task graph and explicit subagent 'ownership' semantics - two unrelated teams shipping supervisory structure the same week, not announcing it. Claude Code v2.1.287 · Zechner, pi
-
SHIPAnthropic ships Claude Sonnet 5.5 and a built-in agent-watcher mod
Sonnet 5.5 runs 30% faster and costs up to 30% less, and is now the model behind Claude's free tier. Claude Code's new 'Mods' system adds a side-agent, 'You should know,' that watches the main agent's work and flags what it might miss. Willison · Claude Code v2.1.287
-
Code review is no longer keeping pace with how much code agents write, and this week's practitioner response was new infrastructure, not better review tools
Gergely Orosz's Oct 6 GitHub data found agent-authored pull requests rose ninefold in eight months and passed human-authored ones in August; Boris Cherny says he runs five to ten Claude sessions in parallel, and a startup engineer told Orosz reviewers now 'stamp everything with LGTM.' The same day, Mitchell Hashimoto shipped a terminal protocol for agents to report status, and Armin Ronacher argued agents need sandboxed code execution, not just CLI calls, to handle that scale of orchestration. Orosz · Hashimoto · Ronacher
-
Multi-agent swarms moved from an argued idea to a shipped product primitive this week
Claude Code added agent.spawn for teammates Oct 3 and an agentType field to subagent status lines Oct 7, a day after Mitchell Hashimoto proposed a terminal protocol for agents to report status. The tooling is catching up to the argument: Ethan Mollick said Oct 1 swarms self-organize without careful human design, and Jack Clark's Import AI the same week framed the trend as 'swarm scaling.' Claude Code releases · Mollick · Hashimoto
-
Mitchell Hashimoto's terminal status protocol (OSC 7501) shipped in Claude Code v2.1.295, October 8. Claude Code v2.1.295
-
The fix for tool-bloated agent context isn't fewer tools - it's a sandboxed scripting layer to call them from
Armin Ronacher's October 6 'Codemode' post argues agents that write and run JavaScript to call tools - batching, composing and stashing results outside the transcript - beat agents that load dozens of tool definitions directly into context, demonstrated in his own Pi coding agent. He says the pattern will spread, but MCP itself needs structured output and better tool discovery to fit it. Ronacher
-
A new terminal protocol went from proposal to shipped support in two days
Mitchell Hashimoto proposed OSC 7501 Oct 6 - a terminal-native protocol letting any program report idle, working, blocked or done status without screen-scraping hacks. Claude Code shipped support for it in v2.1.295 two days later, one more case this year of teams getting more out of improving the harness around a model than out of waiting on the next one. Hashimoto · Claude Code v2.1.295
-
Verification is replacing code review, not improving it
Geoffrey Huntley argued across five posts this week, Oct 7 to Oct 13, that hand-typing code is obsolete and human line-by-line review can't scale with agent output - closing with 'There's no place in the world now for a software engineer who types code by hand.' Simon Willison amplified Huntley's own pre-commit hook, which fails any commit that bypasses an approved diagnostics library, the same day. Huntley, agency · Willison, quoting Huntley
-
SHIPAnthropic previews Flows, letting a lead Claude agent write its own multi-agent coordination
Instead of a fixed orchestrator config, a lead agent in Claude's new Flows writes the program that coordinates its subagents. Anthropic engineers say the fix for runaway parallelism is better evals and lower defaults, not more structure. Latent Space
-
SHIPClaude Code adds idle-conversation compaction and new AWS, telemetry hooks
Version 2.1.299 compacts conversations left idle near the one-hour prompt-cache limit; 2.1.300 adds an /advisor command on Claude on AWS and OpenTelemetry attributes for tracing agent runs. Claude Code releases
-
Verification is replacing review inside the lab that makes the tools, not just in theory
Simon Willison highlighted Anthropic's Oct 16 account of its own Claude Code team: Claude writes about 90% of Claude Code's codebase, and review now 'runs continuously, more like spellcheck while you write' - most changes are reviewed and fixed by Claude before a human looks. Merged pull requests more than doubled from June to September while the share needing a revert fell over 40%. Osmani, Lunt & Wang, Anthropic · Willison, quoting the Claude Code team
-
SHIPClaude Code ships idle compaction, gzip requests and subagent forking by default
Versions 2.1.299 through 2.1.302 compact idle conversations before the one-hour cache limit, gzip large API requests, let organizations pick the auto-mode classifier model, turn on fork subagents by default everywhere, and cap image reads at 128MB. Claude Code releases
-
SHIPClaude Code fixes a worktree bug that could silently delete work
Versions 2.1.303-304 fixed claude --worktree deleting a worktree and its branch without confirmation when HEAD was detached, and added a decider:"person" hook so only a human can approve a sensitive request. Claude Code releases
-
SHIPClaude Code fixes a gateway bug that broke every request on Vertex and Bedrock
Version 2.1.305 fixes a 2.1.304 regression where a gateway rejecting tool search failed every request outright; Claude Code now retries without it. The release also fixed new worktrees sometimes branching from stale local state instead of the default branch. Claude Code releases
-
Anthropic is turning agent safety stops into telemetry data, not just confirmation prompts.
Claude Code v2.1.307 (Oct 21) added a claude_code.safety_event OpenTelemetry signal firing whenever a tool call is stopped or flagged by permission rules, auto mode, safety checks, hooks, or the sandbox - one release after v2.1.305 shipped a Linux sandbox flag whose own changelog warns it can bypass file protections. Claude Code releases
-
ACTClaude Code 2.1.309 closes a real sandbox escape
Version 2.1.309 fixed sandboxed Bash commands running shell startup files outside the sandbox on macOS and Linux, and closed a way sandboxed commands could modify the --mcp-config file itself. Claude Code releases
-
Claude Code v2.1.310 extended its Monitor tool to Bedrock, Vertex and Foundry and fixed a long-standing compaction bug. Claude Code releases
-
AI-generated code volume is outpacing the tools and processes built to review it, and nobody can yet prove the spending is worth it.
Geoffrey Huntley's Oct 15 post argues GitHub's pull-request model is breaking under AI-written patch volume and proposes letting people remix code via Nix overlays instead of submitting PRs. Gergely Orosz's Oct 20 survey of 27 companies found weekly PRs per developer up 113% to 128% at some firms, AI spend per developer varying over 20x with no size correlation, and most companies still unable to show real throughput gains. Geoffrey Huntley, a new model for open source · Gergely Orosz, the ROI of AI spending
-
Some practitioners are shifting from directing agents step by step to handing them a loose goal and letting them orchestrate their own subagents.
Ethan Mollick's Oct 20 essay describes building that post's own charts by giving a Claude orchestrator agent only loose direction - 'make it better' - and letting it spawn and manage sub-agents for research, graphing and citations. He calls this a third mode beyond the 'centaur' and 'cyborg' framing he'd used before: 'conjurer' - summon agents, direct loosely, evaluate the output. Ethan Mollick, 30 Months on the Exponential
-
Dan Luu: AI coding has cut the cost of ambitious side projects, citing a Rust Postgres rewrite that found real bugs. Dan Luu, We should be more ambitious
-
Mario Zechner shipped Explainer, a coding-agent skill turning Markdown into interactive single-file HTML tutorials, built '100% by agent.' badlogic/explainer
-
Claude Code v2.1.311 now runs ultrathink at xhigh effort by default and fixes subagents losing MCP tools that connect late. Claude Code releases
-
Claude Code v2.1.312 drops the 'session limit' name for the clearer '5-hour limit' and caps hooks at 24 concurrent per session. Claude Code releases
-
Agent harness design is converging on a durable, server-side 'brain' coordinating disposable sandboxes for the risky work, with credentials kept outside the agent entirely.
In an Oct 27 Latent Space interview, Anthropic's Katelyn Lesse and Angela Jiang described a durable harness coordinating sandboxes that restart freely, with agents seeing only a placeholder credential a proxy fills in on egress. The same week, Gergely Orosz's Oct 27 survey of AI-SRE systems at Monzo, OpenAI and Stripe found teams independently adding a verifier role 'structurally independent of investigators to avoid self-confirmation.' Latent Space, How Anthropic ships Claude with Claude · Gergely Orosz, AI SRE
-
GitHub's Octoverse 2026: merged pull requests tripled this year to 220 million a month; Simon Willison credits agentic engineering. Simon Willison, GitHub Octoverse 2026
-
Claude Code v2.1.309 closed two sandbox-escape paths: shell startup files running outside the sandbox, and edits to its own --mcp-config file. Claude Code releases
-
Claude Code v2.1.314 adds a setting blocking writes outside the working directory, plus idle compaction in VS Code and Desktop. Claude Code releases
-
Anthropic and Theorem used Claude to prove RISC-V assembly-level security properties across 100+ open-source libraries. Zvi Mowshowitz, AI #192