An Agent Said Verified: A Comment Loop, the Wrong Directory and a Commit the Engine Deleted

My agent fleet is turning into a standalone package called Sovereign Fleet. Three days ago I wrote about the review of its 0.4.0 release, where a sandbox existed only as a claim. This is the next round of the same story, and it is the more useful one, because nothing here was a lie. Every claim had a test behind it. The tests just did not test the thing.

The work order went to Antigravity, one of the agents that coordinate the fleet: build an issue intake (tasks from Gitea and GitHub issues), a daily lane health check, and fix three bugs found in live runs. The report came back the same evening. Every task implemented, 179 tests green (up from 158), each fix “verified”. Claude reviewed it against the code and a live Gitea instance before anything was switched on.

What the report said and what was true

None of this needed cleverness to find. It needed running the thing once against real input.

The comment loop

Gitea’s issue API has an updated_at field, and it looks like the natural key for “has this issue changed since I last looked?”. It is not. A comment counts as an update:

before 2026-10-16T19:57:48Z
after  2026-10-16T19:57:50Z   # after posting one comment

The plugin queued an issue, commented “queued as task-…”, and on the next poll saw a new updated_at. Against a throwaway repository, one issue, three polls: three tasks and three comments. On a real repository with a five-minute poll that is 288 tasks a day per issue, each one a prompt for an agent with write access.

The rebuilt intake keys on the issue itself: one issue becomes one task, ever, whatever happens to the issue afterwards. Only a rejected spec gets a second look, and only when the issue’s title or text changed, which a content hash tells reliably and a timestamp does not:

@property
def revision(self) -> str:
    """Changes with the title or text, not with comments or labels."""
    return hashlib.sha256(f"{self.title}\0{self.body}".encode()).hexdigest()[:16]

Trust got the same treatment. An issue is a prompt, so the intake trusts nobody by default: only an issue whose author is listed for that project and that carries a valid fleet block (the task’s allowed paths and its test) runs unattended. Everything else waits for a human to approve it in the dashboard. A project with no listed authors, which is what a public repository should have, never runs anything on its own.

The wrong directory

The live test before this round had shown opencode on the local Qwen model exiting cleanly and changing nothing, after which aider made the fix. The report traced it to a missing tool_call capability in opencode’s model config and added it. Plausible: I have written before about Qwen’s tool calling, and the bake-off in September found exactly that failure.

The review ran opencode directly inside the package’s harness and read what it said:

✗ Read /data/projects/sovereign-fleet/calc.py failed
Error: File not found: /data/projects/sovereign-fleet/calc.py
✱ Glob "**/calc.py" 0 matches

That is the package’s own source directory, not the task’s worktree. The harness started opencode with the worktree as its working directory, but opencode takes its project from the PWD environment variable, and that variable was inherited from wherever the fleet had been started. The grid runner the package came from never hit this, because it passes --dir explicitly. The fix is one line for every CLI:

# Some CLIs (opencode) take their project directory from $PWD, not from the process's
# cwd: the inherited PWD would point them at the directory the daemon was started in.
run_env["PWD"] = str(Path(worktree_dir).resolve())

The test for it was wrong first. It ran sh -c 'printf %s "$PWD"' and passed without the fix, because a POSIX shell corrects a PWD that does not match its working directory. opencode does not. The test now reads the variable from Python, and it fails without the line and passes with it.

The commit the engine deleted

With opencode finally in the right directory, the first live issue went through: the intake queued it, the router picked the local Qwen lane, the gates passed, and the task reported success. With no changed files, no branch and nothing landed.

git fsck found the work as an unreachable commit:

761de35 | fix: add() returns a + b instead of a - b

opencode had fixed the bug and then run git commit itself. The engine judged the agent’s work by comparing the worktree with HEAD, saw no difference, and since a run that changed nothing leaves no branch behind, it deleted the branch, and the fix with it. Before the directory fix this could not happen, because opencode never touched the worktree at all. One bug had been hiding another.

The engine now treats agent commits as what they are: changes. Before every inspection it puts the worktree back the way it expects, moving HEAD to the task branch if the agent switched branches and soft-resetting to the commit the task started from, so the agent’s commits become staged changes that scope check, gates and the fleet’s own commit all see:

code, head, _ = await _git("-C", worktree_dir, "rev-parse", "HEAD")
if code == 0 and head.strip() != base:
    logger.warning("%s: the agent committed itself; its commits are folded into the worktree",
                   worktree_dir)
    await _git("-C", worktree_dir, "reset", "-q", "--soft", base)

A second guard closes the class of bug: a task never passes with no change left in the worktree. Both tests fail without the fix.

The grid runner, which runs my production tasks, had a milder form of it. It saw a clean worktree after a self-committing agent, logged “changed nothing” and handed the finished work to the next CLI. Its journal shows that message seven times since mid-September. How many of those were self-commits cannot be told after the fact. The runner now folds agent commits back the same way.

What the health check found on its first night

The rebuilt check probes every lane once a day: a tiny chat request with the lane’s real key for API lanes, the model server for local lanes, the model list for opencode lanes. It found four dead lanes, and none of them would have shown up any other way until a task failed on them:

• Two opencode lanes whose free models are gone. One of them vanished overnight, between the evening check and the morning one.
• An NVIDIA NIM lane answering HTTP 410 for Kimi K3.
• A NIM lane whose model is listed in the public catalog but answers 404, “function not found”, for this account.

A dead lane leaves the router’s ranking until a later check passes. A local model that is simply not loaded is not dead: the router checks that before every run and can load it on demand.

The successor that would have been a downgrade

The report on those four lanes ended with a suggestion: replace Kimi K3 with K2.6, because NIM lists K2.6. I asked why an older version should replace the newer one. It should not, and the reasons were all available:

moonshotai/kimi-k3    HTTP 410  "has reached its end of life on 2026-10-15T08:00:00Z"
moonshotai/kimi-k2.6  HTTP 404  "Not found for account"

K2.6 is older (April against July), and models.dev, the open model catalog opencode itself reads, files it under another family and marks it deprecated. A model list says what a provider still advertises, not what still answers. This mattered because I did not want to edit fleet.toml by hand every time a free model disappears; I wanted the fleet to handle a model that is definitively gone. That only works if “definitively gone” and “replaced by” follow rules a model list cannot fool.

So the health check now separates dead from retired. Dead is a rejected key, used-up credit or a 404 for a model the provider still lists; a later check can bring it back. Retired is the provider saying the model is gone for good: HTTP 410, or a model that left the provider’s own list and that the catalog marks deprecated (or that stays gone for a week). Mistral goes further and names the successor in its model list (deprecation, deprecation_replacement_model); OpenRouter publishes an expiration_date. A model with an announced end gets a notice while it still works.

A retired lane takes a successor on its own, under five conditions: same provider, same family in the catalog, not older, not deprecated, not more expensive. Then the candidate has to pass the same probe as the lane itself. The lane keeps its name and settings, fleet.toml is not rewritten, and the router forgets the old model’s history, because a pass rate measured on Kimi K3 says nothing about its successor.

On the first run of the new check, none of the three retired lanes had one. The two opencode models are deprecated with their whole free family, and Kimi K3 has no newer free Kimi on NIM. So they left the ranking for good, which is the correct outcome and the one a hand edit would have gotten wrong. The same run found six free, tool-calling models that no lane used yet and that passed a probe. With routing.discover they become lanes of their own, and the router learns them like any other, as the grid runner’s live roster sync already did for opencode’s free models.

The end-to-end run

The run that counted, against a throwaway Gitea repository with a deliberate bug: an issue with a fleet block from a trusted author, queued on the next poll, picked up by the local Qwen through opencode (which committed on its own again, and this time kept its work), gates green, fast-forwarded and pushed, then a comment on the issue with the commit, and the issue closed. An issue without a block waited for approval and went the same way after it. A broken block got exactly one comment, and after the author fixed it, exactly one task.

Switching it on for real repositories

A throwaway repository runs the package’s own engine. My real repositories run through the grid runner, the older system the package fronts until it can replace it. Switching the intake on for six of them meant two systems had to agree on every field, and they did not. Each side had passing tests; none of those tests ran the other side.

• The gate. A fleet block names the test that proves the task. The intake passed it to the runner as a string; the runner only accepts an object, {"pytest": "<path>"}. The intake in turn rejected that object, the runner’s own documented format. Either way a block with a gate could never have been queued.
• The approval. An issue from anyone but a listed author waits for approval. The intake created such a task as pending and moved it to “awaiting approval” right after. In the mode my runner runs in, it starts pending tasks within seconds, so a stranger’s issue could have run in that gap. The runner now creates the task held, and the intake never sees it pending.
• A lane nobody named. The runner insists on a lane in a task that skips triage. A block without one was refused. The package’s router now picks among the lanes the runner has active, ranked on the runner’s own history, and writes every pick to a log.
• The runner’s own sync. It read the same repositories. It now skips every repository the package’s intake reads, so no issue becomes two tasks.

Then the first real issue, on the package’s own repository: unit tests for two helpers of the model catalog, one allowed file, one test as the gate. The router picked the local Qwen. Eight minutes and three lanes later the task was dead, and no agent had done anything wrong:

ConfigError: ~/.config/sovereign-fleet/fleet.toml: [Errno 13] Permission denied

The repository gate ran the package’s test suite inside the runner’s sandbox. The suite’s first import read the operator’s real configuration file before any test fixture could point it elsewhere, and the sandbox refused, as it should. The runner blamed each lane in turn and escalated. The fix isolates the configuration before the first import. A few tests that need a local test server or a writable home directory now skip inside a sandbox with the reason; in the runner’s sandbox the suite reads 227 passed, 20 skipped.

Sent back to the local Qwen, the task took under a minute: nine tests covering every case the issue listed, both gates green, pushed. The comment on the issue said “passed its gates” and the issue stayed open: the runner calls a successful push success, and the intake only knew pushed and merged. After that fix the issue reads “Landed on the default branch” with the commit, and it is closed.

What the benchmark found in the router

The router picks a lane for each task from measured results. Before trusting it with real repositories I gave it a benchmark: a small order-processing package with eight tasks, each with an acceptance test the agent may not edit. Three routes ran every task, with a 25 minute cap per task (shorter than the runner’s own limit per lane, so a hung task had no time to move on): the router, the router with the local Qwen marked as busy (so it had to pick something else), and the older dispatcher’s keyword rules.

On eight small tasks all three did well. The router sent every one to the local Qwen and landed 8 of 8, first try, in under a minute each. Eight harder tasks went differently (the last row is the run after the fixes below):

The router had sent half of the harder tasks to a fast cloud lane that had passed 7 of 7 small tasks. It timed out all four times, and the router kept choosing it. After fourteen timeouts across the runs, its record still read 7 of 7.

The cause was in how the router read the runner’s task history. A task that failed or was stopped often carries no lane in its result, and a task that ended in the dead-letter queue carries no result at all. Those attempts were dropped. A task the coordinator had rescued by hand after its lane hung counted as that lane’s success. And the benchmark’s own small tasks were part of the history. A second lane looked like 7 of 8; its real record on real work is 5 of 24.

The runner keeps its own statistics for the dashboard, and they were wrong in the other direction: it only counted an attempt as passed when the task landed in full, so every pull request waiting for review and every push that failed after green gates went down as the lane’s failure. The dashboard showed the local Qwen at 32 %, the router believed about 80 %, and the log says 29 of 53, 55 %. Each side had tests, and each side’s tests passed.

Both now read the same thing the same way: each attempt and its outcome from the log, passed when its gates went green, failed when the gates were green on a branch with no commit, not counted when the infrastructure failed under it. Benchmark tasks stay out. On the live history the two counters now agree for every lane.

Honest numbers exposed a second flaw. Every lane that had tried feature tasks was below 50 % there, and a lane with no record started at 50 %, so real feature work would have gone to whichever lane had never been tested. A new lane now starts at the fleet’s average for that kind of task: it gets tried before a lane that does worse than average, never before one that does better.

With the corrected history the router sent all eight harder tasks to the local Qwen and landed 6 of 8, five on the first try, in a median of two minutes: three times what it landed before, and one more than the dispatcher. The two misses went to the runner’s own escalation, which picked the next lanes by its ladder: the lane with the 5 of 24 record first, then one whose daily quota was used up, and once a subscription lane. A benchmark task now never escalates to a subscription lane.

One run on the way there was accidental. Started before the second fix, it sent every task to the lane with the 5 of 24 record, and that lane landed 6 of 8, most in about a minute. On the benchmark it was the best lane I have measured; on real repositories it fails four times out of five. A benchmark tells you how a lane does on the benchmark.

What I take from it

The agent was not careless in the way a person is careless. It wrote tests for what it built, and they passed. They tested its own understanding, and its understanding was where the bugs were: updated_at means “content changed”, PWD cannot matter when the working directory is set, a clean worktree means no work. Every one of those is a reasonable assumption that one real run disproves.

So the rule for the next handover is short: a fix is verified when it has run against the real thing, and a test is useful when it fails without the fix. In this round one test passed for the wrong reason until it was rewritten, and a runner test went red overnight because it depended on which models opencode happened to offer for free that day. The deleted commit was found by neither. It took one live run.

The second half of the round adds a corollary. Where two systems meet, a test on each side proves nothing about the seam. The gate format, the approval gap and the status word were each correct by the tests of the side that wrote them. One real issue on a real repository found all three in an afternoon, and so did one question from me about version numbers. The router’s history was the same story told with numbers: two counters, each tested, disagreeing by a factor of two and a half about the same lane, until a benchmark put the router’s choices next to their results.