Harness over model
Teams are getting more out of improving the harness around a model than out of waiting for the next model.
-
Harness engineering is moving in-house
Up from the hobbyist end of this thread filed August 25. Gergely Orosz, August 25: Ramp built its own agent, Inspect, because engineers “wanted to run more than a few agents in parallel”; it now authors 75% of merged PRs. Two papers submitted August 24 and 25 freeze the base model, evolve only external memory and verification, and report +10 points on Terminal-Bench 2.0. Orosz
-
Harness investment now pays because the model curve stopped absorbing sloppiness
This moves the harness thread filed Tuesday from who builds one to why anyone should. Drew Breunig, quoted by Simon Willison on August 23: “Prior to Fable, it felt silly to waste too much time improving your coding harness or context strategies. A new model would arrive at the same price (or cheaper!) and paper over most of your problems.” Willison
-
The bottleneck moved from writing code to knowing what the agents wrote
This moves the Ramp harness thread filed Wednesday from who builds the agent to who reads its output. Thorsten Ball, August 30, argues for "the spot right in the middle, where speed and defects are in balance," shortening bug lifetime instead of chasing zero. Gergely Orosz, August 25: Inspect now authors 75% of Ramp's merged PRs, reported as a win. Both argue; neither measures review capacity. Ball · Orosz
-
New agent tooling is shipping harness-agnostic by default, not built around one vendor's CLI
Hamel Husain's evals-skills shipped Codex plugin packaging August 31, keeping installation "agent-neutral." Shreya Shankar's new DocWriter, built through August with CMU and UC Berkeley, connects Claude, Codex, Cursor or Pi as the drafting agent. DeepSeek Harness v0.1.2-alpha (August 27 and 30) runs Claude Code and Codex as installable subagents inside its own orchestration. All three demonstrated, not announced. Husain · Shankar · DeepSeek Harness
-
SHIPMeta graduates Muse Code out of beta with an SDK and subscriptions
August 31: inter-session messaging, workflow orchestration for parallel subagents, and a TypeScript SDK in developer preview that talks to a local muse host with no server or network involved. Three subscription tiers now undercut pay-as-you-go Meta Model API pricing. Meta
-
Code review is moving from a human gate to something the harness does for itself
Richard MacManus at Latent Space, September 1: Vercel's AI SDK, Astro and tldraw now replace external pull requests with agent "software factories" that triage, fix and review before a human merges — Vercel's factory authors 25 to 35% of its own merged PRs. Gergely Orosz's five-year retrospective the same day names code review "going optional" as the shift he expects through 2026. A Show HN project found one vocabulary cluster in Claude-authored PRs grew from 0.7% to 39% of a 461,000-PR corpus between early 2025 and mid-2026. MacManus · Orosz · Abraham
-
The gap in AI evaluation is a neglected discipline, not missing tooling
Hamel Husain's September 3 post extends his September 2 finding that automated judges catch most failures but miss the rest: he names five recurring pitfalls, from unvalidated LLM judges to synthetic test sets ungrounded in production data, and argues teams skip "looking at the data," the highest-ROI step, because it doesn't scale the way a tool does. Husain
-
Developers building coding-agent tools keep converging on the same designs because the same models are shaping all of them
Armin Ronacher's September 5 essay Latent Powers argues large models are quietly steering what developers choose to build, pointing to multiple people independently landing on similar projects. His own agent-tooling project, pi, shipped GPT-6 Astra support the day after OpenAI's launch — a demonstration of how fast that convergence moves, not just an argument for it. Ronacher · pi releases
-
Thorsten Ball argues engineers scrutinizing AI-written code line by line may be acting from habit, not from finding real defects. Ball
-
Three independent voices this week say line-by-line pull-request review no longer matches how code actually gets written
swyx's September 1 report found Vercel's AI SDK merges 25 to 35 percent of PRs from its own agent factory, while Astro, tldraw and Flue auto-triage or auto-close external PRs. Thorsten Ball, quoting Rachel Laycock, warns dropping review loses mentoring and shared ownership. Dex Horthy says his team's agent-only 'lights-off' pipeline rotted a codebase in three to six months — agents are rewarded for passing tests, not preserving design. swyx · Ball
-
An HN benchmark of 10 model/harness pairs found Qwen3.8-27B plus OpenCode fastest; a Claude variant sharpest but five times slower. Hacker News
-
Specialized testing techniques agents are told to use — TDD, formal verification, property-based testing, fuzzing — mostly perform worse than no instruction at all
Dan Luu's September 8 study ran 26 testing conditions, about 80 runs each, on Rust reimplementations of Zstd using GPT-5.6 Sol coding agents. Telling agents to do TDD produced more tests of worse quality; formal-methods agents proved properties irrelevant to the actual bug; fuzzing succeeded in roughly 5 of 160 runs. A hand-tuned custom skill did outperform the default, showing specific guidance can work where generic instruction can't. Dan Luu
-
Gergely Orosz: GitHub PR volume is up 5x in three years; risk-based review lifted Duckbill's merge rate 94%. Orosz
-
Code review is being redesigned around AI output, not just overwhelmed by its volume
Gergely Orosz's September 9 interview with OpenAI Codex lead Tibo Sottiaux moves past his own prior PR-volume numbers: Codex was deliberately built in Rust for long-term security despite weaker model support, and review is shifting to pre-code conversations about intent, agents handling correctness checks after. Sottiaux: architecture changes that 'used to take years can now take days.' Orosz · Orosz on PR volume
-
Armin Ronacher is now committing code to Mario Zechner's agent tooling repo pi-mono, not just writing about it. pi-mono
-
A single-source complaint about Astra's code quality now has a full technical case behind it
Armin Ronacher's September 7 post argues Astra's code is aggressively optimized for token efficiency, not human readability, and degrades without close oversight during long unsupervised runs — the reasoning behind an X post filed here September 9, where he called reverting to GPT-5.6 for daily work 'the first time I feel like this is a genuine regression.' Ronacher
-
Hacker News compared nine coding harnesses on a laptop: llama.cpp answered instantly, opencode took 20 minutes. Hacker News
-
Boris Cherny, via Willison: production Claude code needs a higher bar than human-written code. Willison
-
Telling a coding agent to follow standard testing methods often makes it worse, not better
Dan Luu's September 7 test of 26 prompting conditions on a real Rust Zstd implementation found agents given no testing instructions scored above average, while explicit TDD produced twice as many tests that enforced incorrect behavior and formal-verification tools like Verus and Lean 4 mostly proved properties unrelated to the actual bugs. Property-based testing was the one structured technique that helped. Luu
-
Claude Code v2.1.269 added claude plugin eval, scoring a plugin's eval suite with a reproducible JSON and HTML report. Claude Code changelog
-
Coding agents are starting to manage their own testing and coordination, not just write code
Thorsten Ball's September 12 newsletter describes routinely spawning isolated agent 'orbs' with Fable 5.1 and GPT-6 Astra that test features independently, generate video proof, and report back — with the lead agent adjusting its own prompts based on what they find, unsupervised. He calls it working at 'a higher altitude' than manual review allows. Ball
-
Agent transparency is becoming the harder problem than agent capability
Simon Willison's September 12 test had GPT-6 Astra chain OpenStreetMap tools into a working running route in 27 minutes — but the code behind it vanished once the conversation compacted, leaving no way to inspect how it was built. His point: agent frameworks need to preserve pre-compaction reasoning, not just final outputs. Willison
-
Jesse Vincent's episodic-memory v1.6.0 added a fifth supported harness and AWS Bedrock summarization, September 8. obra/episodic-memory
-
Claude Code v2.1.267 fixed a path-traversal bypass in marketplace plugin installs, September 9. Claude Code changelog
-
Harness engineering is consolidating around shared open code, not just competing essays
Armin Ronacher and Mario Zechner, previously known for separate writing on agent tooling friction, both shipped commits this week to Earendil's open-source 'pi' coding-agent harness — Zechner documenting its task-execution subsystem September 9 to 11, Ronacher cleaning up Codex WebSocket handling September 14. Demonstrated, not argued. Zechner commits · Ronacher commits