goose vs aider vs Pi vs opencode on a Local Qwen: 192 Runs and Time to a Correct Result

Counting how many tasks an agent passes tells you little about what it costs to let that agent run unattended on a local model. Across 192 runs on the same hardware, aider passed 39 tasks while goose passed 38, but aider burned 104.5 minutes on failed attempts while goose achieved 1.6 minutes per correct result by failing fast.

Quick Take: In a 192-run bake-off on an NVIDIA DGX Spark with Qwen3.6-35B served by vLLM, aider finished 39 of 48 tasks and goose finished 38 of 48 tasks, but goose delivered correct code in 1.6 minutes per correct result versus 3.0 minutes for aider, spending only 27.7 minutes on failed runs compared to 104.5 minutes for aider. Pi reached 28 of 48 correct results without a shell tool, spending 117.0 minutes on failed runs. Opencode landed only 2 of 48 correct results, most likely because the model wrote its tool calls in a shortened XML form that vLLM’s parser did not pick up: 33 of its runs changed nothing. For an unattended fleet, throughput depends on failing fast rather than retrying to the timeout.

Hardware, Sandboxing, and the 192-Run Automated Harness

When you deploy coding agents to work without a human watching the terminal, the harness architecture matters just as much as model intelligence. I ran every test on an NVIDIA DGX Spark with the GB10 chip and 128 GB unified memory. The entire bake-off used one local model: Qwen3.6-35B served by vLLM. To handle tool execution and reasoning cleanly, vLLM was started with --tool-call-parser qwen3_xml, --reasoning-parser qwen3 and --enable-auto-tool-choice.

I evaluated four distinct agent harnesses: goose 1.55.0, aider 0.86.2, Pi 1.3.0, and opencode 1.19.3. To replicate production conditions, each agent ran unattended inside a Linux Landlock sandbox with exact command invocations:

• goose: goose run --no-session --no-profile --with-builtin developer --max-turns 80 --max-tool-repetitions 5 --text <task>
• aider: executed with --yes-always pointing at the OpenAI-compatible endpoint
• Pi: pi -p <task> --tools read,edit,write --model local-qwen/qwen3.6-35b
• opencode: opencode run --auto -m local-qwen/qwen3.6-35b <task>

Pi ran with only read,edit,write tools and no shell tool, meaning Pi could not run pytest or inspect terminal error output on its own.

The test suite consisted of 16 tasks against a small Python fixture package that was rebuilt fresh for every run. The tasks were divided evenly into two difficulty levels. Tier 1 contained 8 small tasks covering bug fixes, small features, a refactor, and input hardening. Tier 2 contained 8 harder tasks requiring wider architectural comprehension, for example a gift card feature across three modules, an inventory reservation leak, an event dispatch refactor, query indexing, and input validation.

Before each run, the fixture committed an acceptance test and defined an explicit list of allowed paths that the agent was permitted to touch. A run passed only if, after the agent: every changed file was inside the allowed paths (scope check), the repository’s own tests passed, and the task’s acceptance test passed. If any gate failed, the harness allowed one repair pass feeding back the failure output. Each attempt had a hard agent timeout of 1500 s.

To keep time of day from favoring one harness, I executed 3 repetitions of all 16 tasks across all 4 harnesses, totaling 192 runs on 2026-10-19. The runs went one at a time, task by task, alternating harnesses, so every tool saw the same time of day.

Overall Benchmark Scores and Tier Performance Across 192 Runs

The aggregate metrics demonstrate that simple pass rates conceal the operational cost of running local coding agents. Over 48 runs per harness, aider and goose performed almost identically in total task completions, but they diverged completely in execution time and failure overhead.

Separating the results by task complexity clarifies where each harness shines and where sandboxing constraints create hurdles:

In tier 1, goose, aider, and Pi all handled the basic bug fixes and hardening tasks reliably, solving 23, 23, and 21 runs respectively out of 24. Once the benchmark moved to tier 2 tasks that touched multiple modules or leaked reservations, aider solved 16 of 24 and goose solved 15 of 24, while Pi dropped to 7 of 24 and opencode solved 1.

Why Minutes per Correct Result Beats Raw Pass Count for Unattended Fleets

In an autonomous agent queue, the key performance indicator is minutes per correct result. This metric is defined as all minutes across all runs of that harness, both passed and failed, divided by the number of correct results achieved.

When an agent succeeds, the execution is fast across the board. The median minutes of a passed run were:

• aider: 0.1
• Pi: 0.5
• goose: 0.6
• opencode: 1.1

Aider had the lowest median duration on successful runs at 0.1 minutes, landing clean patches quickly when the model grasped the solution on its first attempt. Aider also finished with 39 passes compared to 38 for goose. However, aider spent 104.5 minutes on its failed runs. Its few failures were slow ones, and they drove its overall average to 3.0 minutes per correct result. In tier 2, aider required 6.9 minutes per correct result.

Goose behaved differently. Goose spent only 27.7 minutes on failed runs. When goose failed, it mostly failed fast. That fail-fast behavior allowed goose to post 1.6 minutes per correct result overall and 3.3 minutes per correct result on tier 2 tasks.

Pi accumulated 117.0 minutes on failed runs, resulting in 5.0 minutes per correct result overall and 16.7 minutes in tier 2. Most of its failures were scope violations, not failing tests (see below).

For anyone running an unattended local fleet, failed runs that spin until the 1500 s boundary starve other jobs waiting in the queue. Goose delivered almost the exact same pass count as aider while freeing the local inference hardware in 1.6 minutes per correct result instead of 3.0 minutes for aider.

Scope Violations and Test Failures: How Each Agent Failed

Examining how runs failed reveals the operational personality of each coding harness:

• goose: 8 scope violations, 1 test failure, 1 other
• aider: 8 test failures, 1 scope violation
• Pi: 15 scope violations, 4 test failures, 1 timeout
• opencode: 33 runs with no change at all, 12 other, 1 scope violation

Aider demonstrated strict boundary discipline, suffering only 1 scope violation across 48 runs. Instead, aider failed because of 8 test failures. Aider modified the designated files, but its edits broke existing package regressions or failed the task acceptance assertion. It kept its focus inside the permitted boundaries even when the logic proved incorrect.

Goose showed the opposite pattern, recording 8 scope violations, 1 test failure, and 1 other failure. On tier 2, several of those runs also edited orderflow/__init__.py to export a newly created class. Because orderflow/__init__.py was not listed in the task allowed paths, the gate immediately marked the run as a scope violation. In other runs, the agent edited the acceptance test or other tests instead of the code. The harness gate correctly blocked those modifications.

Pi accumulated 15 scope violations, 4 test failures, and 1 timeout. My fleet runs Pi without a shell tool, so Pi could not run pytest before finishing. Most of its failures were edits outside the allowed paths, the same orderflow/__init__.py export and test edits seen with goose.

Opencode had 33 runs with no change at all, 12 other failures, and 1 scope violation. The next section is about why.

The Opencode Anomaly: When vLLM XML Tool Parsing Drops Tool Calls

Opencode recording only 2 correct runs out of 48 looked like an outright collapse, so I wrote a small reproduction script to see what it actually printed.

In a separate 10-run reproduction test, opencode printed its tool calls as plain text in 9 of 10 runs. For example, it output <read><parameter=filePath>, which is a shortened variation of Qwen’s XML tool call format. vLLM was running with --tool-call-parser qwen3_xml, the parser the vLLM tool calling documentation lists for Qwen’s Coder models. That tool-call parser looks for the full <tool_call><function=...> form.

Without the <tool_call><function=...> wrapper the parser does not recognise the text as a tool call, so it reaches opencode as ordinary text and nothing is executed. That is the most likely explanation for the 33 runs that changed nothing, although I only confirmed the pattern in the separate 10-run test.

This behavior represents a configuration incompatibility between this model, this vLLM parser and opencode’s prompt in my setup. It is not evidence of poor agent logic in opencode. In fact, during the same night, opencode ran against this exact local model and built a real feature of my fleet’s task runner, cancelling a running task, correctly in about 13 minutes. Similar reports in opencode’s issue tracker were closed as model problems.

Goose in Production: Crash Recovery, Clean Profiles, and the Two-Second Exit

While goose achieved the highest operational efficiency, it displayed two distinct rough edges during testing.

The first issue occurred on one long task specification during the early morning. In 2 of 3 tries, goose terminated after approximately 2 seconds with exit code 0 and zero tool calls. The local Qwen model narrated its execution plan in text and generated pseudo tags such as </turn-context>. With no tool call in that answer, goose took it as the final answer and ended the session without running anything. Across the entire 192-run bake-off, this behavior occurred at most once. In 14 later valid runs executing against that identical long specification, goose made active tool calls in every run, ranging from 14 to 72 tool calls each. Because the error was not reliably reproducible, I have not submitted an upstream bug report.

The second issue was an initialization panic. Goose crashed once at startup due to an internal sqlx database panic in its session storage, printing index out of bounds: the len is 29 but the index is 29. The subsequent retry succeeded, and 0 of 20 later runs reproduced the crash. To prevent session state corruption from breaking unattended queues, my fleet runner now starts goose with --no-session and automatically triggers a retry if the process panics within 15 seconds.

This differs from my earlier 3-task test on 2026-09-23, where goose scored 2 of 3. In that test goose had loaded my personal profile with 20 extensions; this time it ran with a clean one (--no-profile --with-builtin developer). I have not measured how much of the difference that explains. For how goose compared with other local coding CLIs before, see goose vs Vibe vs opencode.

What Unattended Execution Means When Gates Catch Broken Code but Not Wrong Code

During the same night as these benchmarks, my fleet used goose on this local Qwen model to write core modules of its own internal task runner. Specifically, goose wrote the loop that runs the queue, the crash recovery and the wiring into the daemon; opencode wrote the cancellation (see above).

Every one of those goose runs passed all gates: the repository’s tests, the task’s test and the scope check. Yet when I reviewed the diffs, I still uncovered critical semantic bugs in the generated code.

In one, the loop marked every task “completed” after the run, overwriting the status the engine had set, so a failed task would have looked successful. In another, the cleanup of worktrees left over after a crash removed them without first committing what the agent had written there, so that work would have been lost. That one came straight from my own task spec, which wrongly said the work was already committed.

Automated gates check syntax, tests and paths. They catch broken code, but they do not catch wrong code. When an agent passes every test in a sandboxed run, human code review remains indispensable before merging changes into production repositories.

Benchmark Caveats and Measurement Limits on a Shared Local Server

To interpret these measurements accurately, several operational constraints must be kept in mind:

• The entire benchmark was evaluated on one model only, Qwen3.6-35B, and different model families may interact differently with each harness prompt template.
• The test suite used synthetic fixture tasks. An earlier router benchmark I conducted indicated that synthetic test packages often rank agent performance differently than real-world repositories with complex build systems.
• The evaluation used 3 repetitions per task. While 192 total runs provide clear trends, edge-case flakiness can still appear across larger sample sizes.
• Pi ran without a shell tool because that matches how my fleet sandboxes it, which prevented Pi from running pytest to verify its own work.
• Opencode’s low score most likely reflects a specific XML tool-call parser mismatch in my setup rather than inherent tool reasoning capability.
• The test runs shared the local model server with a few background review jobs at times, which may have slowed some runs.
• Twice during the test sequence, an active run of Pi overlapped a model-server restart triggered by its background watchdog.

What I Am Changing in My Local Agent Fleet Routing

These 192 runs gave me concrete numbers to optimize my unattended agent fleet. Moving forward, I am implementing three specific adjustments:

First, I will run goose and aider as separate lanes in the fleet and let the router keep choosing between them by measured pass rate and time, instead of me picking a winner from one benchmark.

Second, the fleet’s engine will treat a run that exits 0 with no change after a few seconds as a failure, not a success. This catches the rare early termination seen in goose and flags prompt formatting dropouts.

Third, I will try another vLLM tool-call parser to see whether opencode’s tool calls then get through.