Autonomous coding agents can land dozens of tasks in a day, but passing tests only proves code executes, not that the underlying claim holds. Here is the operational reality of our review loops, benchmark traps, and the decision to freeze development.

The Fleet Kept Finding Bugs. So I Froze It.

On 2026-09-24, my coding agent fleet landed 25 tasks. Every one of them had passed its test gate. Almost every code task among them still needed a follow-up fix once a reviewer read it. The test suites exited zero; the claims behind them did not always hold.

Quick take: The fleet was not failing because language models could not write syntax or test runners had stalled. It hit the hard limit of automated verification: gates prove that code runs, not that an engineering claim holds true. Our 7-day quality metrics on 2026-09-25 recorded 45 landed tasks out of 66 finished work tasks (68 %), with 64 % landing on the first try and a median of 52 minutes per landed task, requiring 0.62 coordinator interventions per task. But adding a closed review loop and benchmarks uncovered deeper issues: silent model fallbacks, blind validation passes, and self-inflicted orchestration bugs. The answer was not adding more tooling. On 2026-09-25, I instituted a complete feature freeze across the fleet and the podcast app I am building, Vozonda, until the app’s public 1.0 release, with an exit criterion for the fleet: two weeks below 0.2 coordinator interventions per landed task and no new bug in fleet code. One week later the freeze had held only in part, and the metric behind that exit criterion turned out to count almost nothing; the last section says what happened.

Autonomous Coordination and the Throughput of 2026-09-24

When I documented the shift to autonomous dispatch in 27 Agents, One GPU: How the Fleet Automated Its Own Coordination, the goal was throughput and clean role separation. We separated the coordinator from execution, isolated worktrees, and addressed the harness bottlenecks analyzed in aider vs goose vs opencode, Measured: The Seven Fleet Bugs I First Blamed on the Models. Using Gitea as Source-of-Truth for AI Pipelines, tasks were dispatched into dedicated branches with verification commands tied to each issue.

The immediate result was volume. On 2026-09-24 alone, 25 of 32 finished work tasks landed. Of those 25 landed tasks, 19 landed on the first try, while 6 landed after escalation to another lane. In parallel, 6 of 6 review tasks landed.

On paper, landing 25 tasks in a single day looks like an operational triumph. But behind those green checkmarks lay a sobering reality: almost every landed code task on 2026-09-24 needed a follow-up fix after review. The gates accepted code that satisfied its own assertions and still did the wrong thing.

Why Verification Gates Catch Broken Code Instead of Wrong Code

The core lesson from that run is simple: gates catch broken code, not wrong code. A test harness verifies execution mechanics: whether files parse, modules import, assertions run, and the process exits zero. It cannot tell whether the change satisfies the actual system requirements.

When an agent writes code to pass a gate, it writes code that makes the test suite exit zero. If an assertion expects a dictionary, the agent returns a dictionary. If an operation raises an unhandled exception, the agent adds an exception handler or adjusts the condition until the gate passes. The gate is green, but the logic can be completely inverted.

Four examples from the tasks that landed on 2026-09-24 show how this happens:

  1. An inverted rule: a check that should warn when a task names no test file warned in exactly the opposite case. Its tests passed, because they had been written against the inverted behaviour.

  2. Raw keys stored where only hashes belonged: the tests checked that a key went in and came back out. They did not check what was written to disk, which was the key itself.

  3. The public-readiness scanner: a scanner meant to find leaked secrets before a public release flagged 2,438 package hashes in a lockfile as secrets. It ran without crashing, so it landed; the real findings were buried under the noise.

  4. The Docker quickstart: its configuration tests were green, but the engine variable it set was read by nothing, so a fresh container would still have asked for a GPU.

The quickstart is also the best example of what finally works: running the claim for real. On 2026-09-25 I built it from a clean clone and used it like a new user would. The first episode request failed, because the API correctly refused writes from a non-local bind and the web UI sends no token. The database path had been dropped from the configuration by a later task, whose own test mistook the container path for a host path. None of this was visible to any gate.

Later that day the same clean-clone run found the last blocker: the voice stage failed inside the container. The text-to-speech library no longer downloads voice models on its own, 6 of the 14 voice model names in the code did not exist upstream, and English episodes were cast with German voices. After the fix, a clean clone rendered a 70-second English episode on CPU. Every one of those bugs had passed its tests.

In each case, the gate confirmed that code executed. It could not confirm that the code was right.

The Benchmark That Measured Nothing

The podcast app has a quality benchmark: five fixed sources, each also rendered by NotebookLM as a reference, and a report that compares the two on script and audio metrics. On 2026-09-25 I opened that report to decide what to tune next. Every single cell showed a dash.

The report script expected a flat metrics format that the metrics script never wrote. Its tests passed anyway, because they used a fixture in that invented format. The numbers were on disk the whole time; nothing read them.

Once the report read the real files, two more measurement errors surfaced:

  1. The phrase filter parsed every backticked word in a table row, including the word “in” from an explanation column. Almost every turn of every script matched, so the “overused phrases” metric was roughly the turn count, for my episodes and for NotebookLM alike. Counted correctly, overused phrases were 0 to 5 per episode.

  2. The NotebookLM references are transcribed without speaker separation, so they split into about 30 chunks of about 120 words. Our scripts have 100 or more real turns. “Questions per turn” was therefore not comparable, and the earlier finding that my scripts asked too few questions was an artifact. Per 1,000 words, they asked as many questions as NotebookLM or more.

What remained real was rhythm. In every script from the local model, nearly all turns ran 15 to 30 words. There were no quick reactions at all, and almost no contractions. NotebookLM sounds like two people talking because turns vary: a long explanation, a “Right.”, a sentence the other host finishes.

Prompt or Model?

I rewrote the dialogue prompt three times and measured each version on three sources at NotebookLM length, generating scripts only.

Version 1 removed a length rule that forced every turn into one word band. The dialogue became more conversational, but turn lengths stayed uniform. Version 2 added a short example exchange on an unrelated topic to show the rhythm. The model copied it verbatim into an episode about the Lightning Network, bakery and all. That broke the one rule the app is built on, grounding in the source, so the example was dropped and a test now keeps worked examples out of the prompt. Version 3 moved a short form check after the source text, closer to where the model writes. The local model still ignored it.

Then I ran version 3 unchanged on two cloud models through NVIDIA NIM, using the free developer key (NVIDIA’s trial terms allow testing and prototyping, not production):

Local Qwen3.6-35B-A3BKimi K3Nemotron 3 Ultra
Quick reactions as their own turn0 to 1 %2 to 13 %9 %
Turn length variation (CV)0.20 to 0.22 (one German outlier at 1.09, from a single 339-word turn)0.40 to 0.620.61
Contractions per 1,000 words (English)about 1.5about 3713.8
Scripts delivered3 of 33 of 31 of 3

Kimi K3 wrote natural dialogue in English and German with the same prompt. Nemotron 3 Ultra was good on the one script it delivered, but twice returned no valid output. The limit was the local model, not the prompt.

The app now offers Kimi K3 as an optional cloud script writer, with your own NVIDIA key; the free developer key is fine for trying it, production use needs NVIDIA’s paid licence. The engine menus are grouped into local and cloud, and a cloud engine says plainly that the source text leaves the machine. The default stays local. (Since 2026-09-29 the endpoint has not served Kimi K3 to my key; see the last section.)

None of that tuning could mean anything until the benchmark itself was fixed: a report that showed nothing, a metric that counted a preposition, and a gap that was a transcription artifact. The benchmark had tests. The tests proved the code ran.

The Silent Fallback and Keyring Failures in the Sandbox

Beyond code logic errors, the fleet suffered from measurement deception inside its own harness. In The Engineering Honesty Manifesto, I committed to reporting verified operational facts. Yet our statistics were distorted by an environmental failure, echoing the traps described in A Benchmark Handed Me a Number Three Times in One Day. Three Times It Was Lying..

Every fleet run on the Antigravity (agy) lanes, 10 of 10, failed inside the sandbox. The cause was subtle: agy keeps its login in the desktop keyring over D-Bus. To isolate agent execution, our runner uses a sandbox built with Landlock and a seccomp filter. That sandbox blocks unix sockets.

With unix sockets blocked, agy could not contact D-Bus to read its credentials. Instead of reporting an error, our runner triggered a fallback mechanism. That fallback quietly ran OpenCode with Nemotron under the agy lane name.

The dashboard showed successful completions credited to agy, while Nemotron via OpenCode had actually done all the work. The statistics credited the wrong model for 10 of 10 runs.

The fix required three actions. First, a one-time login inside the same sandbox, allowing agy to store its token in a file. Second, replacing the silent fallback with an honest [infra] failure. Third, rebooking the wrong statistics through a recorded, reversible correction instead of hand-edited logs.

Closed Review Loops and the Seeded Bug Benchmark

To prevent incorrect code from reaching the repository, we introduced a closed review loop: every landed feature is reviewed by a different free lane. The review ends with a machine-readable findings block, and high findings automatically become a fix task.

To measure how well these reviewers perform, we applied the method from Agent-bench: stop trusting install counts, start measuring your agent’s tools and built a reviewer benchmark.

The benchmark fixture presented a commit claiming to be a harmless refactor that hid 3 seeded bugs:

We ran the benchmark across 4 tested lanes:

The conclusion was direct: the reviewers find bugs when they look. When presented with code diffs, they spot boundary flaws, truncation, and currency bugs reliably. What is missing is someone running the claim for real. Reviewers inspect diffs; they do not verify that a deployed service behaves as intended in production.

Coding Benchmark Round 1 and Baseline Execution

Alongside review tests, we established coding benchmark round 1. The fixture is small, about 250 lines, functioning as a floor check rather than a model ranking. It tests whether a lane can receive a bounded specification, produce valid syntax, and pass deterministic gates.

The measurements on this baseline were consistent:

The benchmark confirmed that basic code generation is reliable across both local and efficient models on small fixtures. When tasks are small and self-contained, the models solve them rapidly. The breakdown in production does not stem from syntax generation. It comes from subtle system interactions and multi-file dependencies where a passing test suite does not ensure functional correctness.

Orchestration Layer Flaws Discovered Under Load

Running the fleet under sustained task volume revealed three bugs in the fleet coordinator itself:

  1. A cancelled task still landed its change: cancelling marked the task, but the running agent finished anyway and nothing checked the cancel before landing. The same change landed twice.

  2. A task hung in “verifying” forever: after a runner restart, only tasks marked “running” were picked up again.

  3. Re-queuing opened duplicate issues: When a task failed a lane and was re-queued, the dispatcher opened duplicate issues in the tracking backlog rather than attaching to the existing issue.

All three fleet bugs were fixed on 2026-09-25. But they proved that our orchestration layer was introducing its own instability into the development cycle.

Our 7-day quality metrics on 2026-09-25 showed the broader trajectory:

A week earlier, the metrics showed:

While landing efficiency rose from 62 % to 68 % and first-try success moved from 61 % to 64 %, coordinator interventions increased from 0.55 to 0.62 per landed task. More code was landing, but keeping it correct required more intervention.

Why do new bugs keep coming? Three forces drive the cycle:

(1) Gates prove code runs, not that a claim holds. As tasks become complex, code passes unit tests while failing the broader design contract.

(2) The fleet changed daily, so each new capability brought its own edge cases. Adding sandboxing rules, new routing paths, and automatic fallbacks created unexpected failure modes like blocked sockets and duplicate issues.

(3) The review loop looks harder, so more gets found. The automated review loop catches subtle logic errors that previously slipped through, generating follow-up work that expands the backlog.

The Feature Freeze and Exit Criteria

By 2026-09-25, the path forward was clear. Adding more features to the fleet was not improving our results. It was creating churn: landing 25 tasks in a day, only to spend the next day diagnosing why green tests hid broken logic.

On 2026-09-25, I instituted a complete feature freeze across the fleet and the podcast app I am building until the app reaches its public 1.0 release.

During this freeze:

For the fleet, the exit criterion is strict: two weeks below 0.2 interventions per landed task and no new bug in fleet code.

Autonomous agents make it trivial to generate code at volume. But software engineering is defined by stability and correctness, not commit frequency. Until our gates can verify that an engineering claim holds true in reality, the most effective decision is to freeze the fleet and stabilize the code we have.

One Week Later: What the Freeze Held, and the Metric That Counted Nothing

On 2026-10-02 I checked the freeze against the commit history. It had held for the fleet’s bug fixing, not for features. Between 2026-09-26 and 2026-10-02 the podcast app received eleven commits labelled as features: new publishing and hosting options, an agent integration, a new default voice engine for the quickstart, a cancel button and script rhythm work. The fleet gained a second local model lane and a rule that hides lanes which have been dead for two weeks. I approved every one of them myself. Each had a reason, but a freeze with a new exception every day is not a freeze, and I would rather say so than let the title claim more than happened.

The same day, the fleet dashboard showed “Landed 0 of 2” for the last seven days. The fleet had landed 28 tasks in that week. The quality metric only counted tasks whose source was spelled claude_code_coordinator; the coordinator had created 37 tasks as claude-code-coordinator and 5 more as claude_coordinator, so nearly all real work was invisible. The metric had a test. The test built its tasks with the one spelling the code knew, so it passed. With the spellings normalised, the week reads 28 of 37 landed (76 %), 64 % on the first try, and a median of 14.6 minutes per landed task.

The number the exit criterion depends on did not survive the same look. It reads 0.07 coordinator interventions per landed task, far below the 0.2 target, but it only counts log lines a coordinator writes into a task. When a review finding is fixed directly in the repository, no such line exists, and this week many findings were fixed exactly that way. So 0.07 does not mean the fleet needed almost no help. It means the metric cannot see the help it got. Until it can, the exit criterion cannot be judged, and because the spelling bug was a bug in fleet code, the two-week clock for “no new bug in fleet code” starts again on 2026-10-02.

Kimi K3, the cloud writer from the benchmark above, has also stopped working for me. The fleet’s daily health check got HTTP 404 for it on 2026-09-29 and switched the lane off. On 2026-10-02 the model was still listed in NVIDIA’s catalogue, but a direct request got no answer within 90 seconds, and Kimi K2.6 answered that the function was not found for my account. A free trial endpoint is not something to build on; the app’s default stays the local model. Whether the local model really is the limit for natural dialogue is not settled either. That gets its own measured follow-up.

The benchmark had tests, and so did the quality metric. Both proved that the code ran.

Two Weeks Later: The App Has a Name, and the Freeze Is Over

The podcast app is called Vozonda now, and it is public: an open-source way to turn articles, papers, PDFs and notes into a podcast on your own hardware, in your own feed. The release article has the details. I am publishing version 0.6, not the 1.0 the freeze was waiting for. A freeze that is lifted for a release it never reached deserves the plain word: it ended, it did not succeed.

The fleet’s part in the last two weeks was smaller than its throughput numbers suggest. Renaming the app touched 147 files. The fleet’s first attempt crashed before it changed anything, and the second one had edited six files after forty minutes when I stopped it and did the rest by hand. Mechanical work across the whole repository is where a single careful pass beats a queue of agents that each see one task.

Two failures from that week changed how the fleet works. A test run reached the production database and reset live settings, because the test setup redirected the old environment variable names but not the new ones. Now the test suite refuses to start when any production path is reachable, and a test proves it. And the fleet’s sandbox has no network at all, which is right for safety and means it can never run a real end-to-end check against the live app. Those checks stay with the coordinator.

The review step earned its place too. An episode made with default settings, which is how the blind test on vozonda.com uses it, says that 100,000 satoshis are about a tenth of a bitcoin; it is a thousandth. I left that episode as it is, because the test compares defaults. For the showcase samples I read every script against its source before it was voiced, and that review caught a claim that more renewable capacity was installed in 2023 than in all earlier years together, which the source does not say. That is the same lesson as the gates above: the model writes fluent text, and something outside the model has to check the claims.

Was this worth it? Zap the article.

Value for value, no signup. Sats go straight to the writer.

… sats zapped
… zaps
Today 7d 30d All-time
Unique readers — — — —
Page views — — — —
—
All Article Insights →