A LoRA Trained for Pi, on a 4-Bit Qwen3.8-27B: It Loads, It Works, and My Tasks Were Too Easy to Tell
On 16 November the harness-hub organisation on Hugging Face published Qwen3.8-27B-pi-adapted, a LoRA adapter for Qwen3.8-27B trained with reinforcement learning for one coding agent, pi. The download contains extra weights, 638 MB under Apache-2.0, that teach an existing base model how the pi harness works.
My agent fleet runs pi as one of its harnesses, the 27B is my on-demand model for hard problems, and the box is one DGX Spark. Testing the adapter on my setup was an obvious step, but there was an immediate catch. The adapter was trained on unquantized bf16 weights, and what fits next to everything else on my machine is NVIDIA’s 4-bit checkpoint.
As in the Flash-Next test, everything up to the results was written before the model answered its first request. I wrote down the arms, six hypotheses with a pass mark each, a stop rule, and the decision rule for becoming a fleet lane. The results were appended afterwards, and the preregistered text remained unchanged. The short outcome was that the adapter loaded cleanly and called tools on the 4-bit base, but my benchmark tasks were solved so consistently by the base model that the test could not tell them apart.
What the makers claim
The model card for revision 80d05c9, the checkpoint I tested, describes the training plainly. The authors used a LoRA with rank 32 and alpha 64 on the q, k, v, o, gate, up, and down projections. Training ran with TRL’s AsyncGRPO trainer, marked as experimental, on 2x B200 accelerators using Harbor tasks served by OpenEnv. The published checkpoint is step 160. Training stopped at step 191 of 1,000 planned.
The card reports paired results under identical settings: one attempt per task, thinking off, pi as the harness, 100 agent steps per task, temperature 0.8, and top-p 1.0.
| Benchmark | Tasks | Qwen3.8-27B | + pi adapter | Newly solved / lost |
|---|---|---|---|---|
| SWE-bench Pro, hard 51 | 51 | 39.2 % | 64.7 % | 15 / 2 (p = 0.0023) |
| Terminal-Bench 2.1 | 86 | 39.5 % | 55.8 % | 16 / 2 (p = 0.0013) |
| Terminal-Bench 2.1, tasks not used to pick the checkpoint | 68 | 39.7 % | 55.9 % | 13 / 2 (p = 0.0074) |
The evaluation datasets are linked directly: SWE-bench Pro hard 51 and Terminal-Bench 2.1. In addition to pass rates, the card states that the adapter needs less than half the turns of the base on tasks both solve. With thinking off, the adapter matches the base model thinking at its highest effort on Terminal-Bench 2.1 (56.6 % against 54.2 %), while generating about 4.5x fewer tokens than the thinking base.
Two details on the card made me trust the report.
First, it corrects itself. An earlier version reported 29.4 % to 62.7 % on SWE-bench Pro. The card explains that nine base model rollouts failed because the tunnel between the sandboxes and the model server went down mid-run. They reran every affected rollout on both sides and published the corrected numbers. A second correction raised the thinking-on scores after a 16k output cap cut long reasoning short. Both corrections remain in the card’s history.
Second, it says where the gain comes from. On SWE-bench Pro, 20 of the base model’s 31 failed attempts ended because it ran out of its 100 model calls, against none for the adapter. The card puts it plainly: “part of the adapter’s gain is finishing within them”, within those limits. That sentence turned out to matter for my result.
The card also warns that the adapter was trained on bf16 weights, that lower-bit quants drift further, and that its GGUF variant has not been re-evaluated. That warning is the reason for this test.
What I can run, and what differs
The unquantized bf16 base is about 56 GB and decodes slowly on a bandwidth-bound box. What I had on disk was nvidia/Qwen3.8-27B-NVFP4, which takes up 21 GB. The label NVFP4 undersells it, because its hf_quant_config.json specifies MIXED_PRECISION. The MLP projections of all 64 layers and the lm_head are NVFP4. The attention projections of the 16 full-attention layers and the projections of the 48 Gated-DeltaNet layers are FP8.
The LoRA sits on two different low-bit formats at once, attaching to FP8 in attention and NVFP4 in the MLP. The makers put no LoRA on the DeltaNet layers, and neither does this setup.
| Makers | Me | |
|---|---|---|
| Base weights | Qwen/Qwen3.8-27B, bf16 | nvidia/Qwen3.8-27B-NVFP4, mixed FP8 / NVFP4 |
| Engine | vLLM with --enable-lora | a community vLLM nightly, 0.23.1rc1.dev309, pinned by digest, same flags as the card, speculative decoding off |
| Harness | pi | pi 1.5.2 inside my fleet’s sandbox, tools read,bash,edit,write |
| Budget | 100 agent steps; 1 h (Terminal-Bench) or 2 h (SWE-bench Pro) per task | 1,500 s per task, no step cap |
| Context | 131,072 tokens | 131,072 tokens |
| Tasks | SWE-bench Pro hard 51, Terminal-Bench 2.1 | the 16 tasks of my harness bake-off |
| Hardware | trained on 2x B200, eval hardware not stated | one GB10 with 121 GiB of unified memory, production model paused |
Three mistakes caught before the first run
The first version of this plan compared the adapter only with my production model, Qwen3.6-35B-A3B, on a handful of fleet tasks. A review before any GPU time found three flaws, and the plan was rewritten before the load. I keep them in because each would have produced a clean, meaningless number.
First, the initial plan had too many variables. Testing the adapter against production changes the model, its size, and the harness at once. If the adapter won, nothing would say why. The fix was to compare the 27B with the adapter directly against the same 27B without it, from one server.
Second, the fleet default had the wrong tool set. My fleet runs pi with read,edit,write only, no shell. The adapter was trained with a shell. Testing it without bash would test something it never learned. The fix was to give both pi arms read,bash,edit,write.
Third, the server defaults had the wrong sampling. Because pi sends no sampling parameters, the server would fall back to the checkpoint defaults of temperature 1.0 and top-k 20 instead of the makers’ 0.8. The fix was to pin temperature 0.8, top-p 1.0, and top-k off for every request using --override-generation-config. Top-k off is my reading of TRL’s GRPO default, where top_k = 0 disables it; the card names no value.
One more check took place before the load. The adapter ships its own chat template. A diff against the base template showed it differs only by the {% generation %} markers that training uses to mask the loss, so the base template was used.
The arms
| Arm | Model | Harness | Server |
|---|---|---|---|
| A, base | Qwen3.8-27B mixed FP8 / NVFP4, thinking off | pi with a shell | the experiment server |
| B, adapter | the same base plus the LoRA, thinking off | pi with a shell | the same server, same process |
| C, the bar | Qwen3.6-35B-A3B, production | goose 1.60.1, the first step of my local fleet lane | production, re-run fresh |
Using --enable-lora in vLLM serves the base model under its own model identifier and the adapter under a second identifier from one process. Arm A and Arm B share quantization, engine flags, sampling parameters, and harness configuration. Only the adapter differs between them.
Arm C answers a different question: is the result good enough to use in my fleet? It was re-run fresh instead of quoted from October because pi, goose, vLLM, and two benchmark tasks had changed since that test.
The instrument
The tasks are the 16 problems from my harness bake-off. These are real fleet tasks divided into two tiers of eight, covering bug fixes, features, refactors, a performance job, and input hardening, with the larger ones placed in tier 2. Each run goes through the fleet engine in its sandbox and passes only if the task’s pytest gate passes. A change outside the task’s allowed paths counts as a failure. The two tasks that were broken in October were fixed a week later and now ship a reference solution.
Task runs were interleaved across the arms so no single arm benefited from a specific time of day.
The bench driver needed one change before the first run. The script originally had pi’s model parameter hard-coded. I updated it to accept named arms, allowing A and B to run as two pi arms targeting two model identifiers on the same endpoint, and its restart check now watches the experiment server. In addition, the fleet writes "api": "openai-completions" into pi’s configuration directory. The model card states that this setting is required. Without it, pi defaults to the Responses API, which the adapter was never trained on.
The hypotheses, written before the run
| Expectation | Pass mark | |
|---|---|---|
| H1 | vLLM serves the LoRA on the mixed FP8 / NVFP4 base | it loads and answers; if not, the failure and its cause are the result |
| H2 | the adapter is not broken | both model ids answer two known-answer questions and pass my forced tool-call check |
| H3 | the speed cost is small | B decodes at least 80 % as fast as A, same server, no speculative decoding |
| H4 | the makers’ gain survives my base and tasks | B passes more runs than A, newly solved minus lost at least +3 per 16-task round |
| H5 | it works with fewer tool calls | on tasks both pass, B needs at least 30 % fewer tool calls (median) |
| H6 | it is good enough for the fleet | B at least level with C in passes, at most 1.25x C’s minutes per pass |
The speed figures I published previously for the 27B, 20 to 21 tok/s, were measured with speculative decoding using MTP and three draft tokens. Hypothesis H3 evaluates A and B without speculative decoding, so those earlier numbers are not comparable with this test.
The procedure: load the server in a quiet window, with no fleet task running and production paused for the whole phase. Checks for H2 and H3 ran on both identifiers. Round 1 evaluated 16 tasks across A and B in interleaved order. If B solved fewer tasks than A in round 1, the protocol required stopping immediately, restoring production, and publishing that result. Otherwise rounds 2 and 3 followed (96 runs for A and B total), then production was restored to run C three times (48 runs).
The decision rule was set before the run. The adapter becomes an on-demand fleet lane only if H4 and H6 both hold. It needs the whole machine, so it would never replace production. If H4 passes without H6, it works on 4-bit but does not beat what the fleet has, so it stays a documented option. If H4 fails, the gain does not survive my base and harness. In that case, merging the adapter into the unquantized bf16 base, a download of about 56 GB that decodes an estimated 2.5 to 3x slower, is the only path left and gets its own decision.
Results
The test ran on 19 November, from 17:20 to 22:32. Every figure below is taken directly from the test run’s result log, covering 96 valid runs for A and B, and 48 for C.
| Pass mark | Measured | Verdict | |
|---|---|---|---|
| H1 | loads and answers | ready after 9.6 minutes, including stopping production and torch.compile; both ids served from one process | pass |
| H2 | known answers and forced tool calls on both ids | 17 x 23 = 391 and Canberra on both; tool-call check 6 of 6 on A and 6 of 6 on B, with thinking off and on | pass |
| H3 | B at least 80 % of A | A 12.4 tok/s, B 11.3 tok/s decode median, prose and code alike: 91 % | pass |
| H4 | net +3 per round | A 47 of 48, B 48 of 48; net per round 0, 0, +1 | fail |
| H5 | 30 % fewer tool calls | not measured | no verdict |
| H6 | level passes, at most 1.25x minutes per pass | passes B 48 of 48, C 45 of 48; minutes per pass B 2.38, C 0.70: 3.4x | fail |
Under the decision rule written before the run, H4 failed. The adapter does not become a fleet lane, and the bf16 merge is not justified by this test.
Where the runs went
| A, base | B, adapter | C, goose + Qwen3.6 | |
|---|---|---|---|
| Passes, rounds 1 / 2 / 3 | 16 / 16 / 15 | 16 / 16 / 16 | 16 / 15 / 14 |
| Passes in total | 47 of 48 | 48 of 48 | 45 of 48 |
| Passed without a repair pass | 36 | 42 | 40 |
| Minutes, all runs | 127.6 | 114.2 | 31.5 |
| Minutes per pass | 2.71 | 2.38 | 0.70 |
| Median minutes of a passed run, tier 1 | 1.05 | 0.95 | 0.30 |
| Median minutes of a passed run, tier 2 | 3.0 | 2.45 | 0.60 |
A’s one failure was a stall in round 3 on the gift-card task in tier 2, producing no output and no change in the worktree for 480 seconds, after which the fleet stopped the run at 12.5 minutes. C’s three failures were all scope violations where goose edited the package’s __init__.py, which the task did not allow, twice on the gift-card task and once on the webhook task. The pi arms had no scope violations in 96 runs.
The stop rule after round 1 did not trigger, as A and B both solved 16 of 16.
Where H4 failed
A solved 47 of 48 runs. A rule asking for +3 per round cannot be met when the base model leaves at most one task per round to win. My 16 tasks are too easy to separate these two models, and that is a statement about my tasks, not about the adapter.
The data does show a direction. B never lost a run A passed. B passed 42 runs without a repair pass, against 36 for A. B was about 10 % faster over all runs, and its median passed run was 10 % faster in tier 1 and 18 % faster in tier 2. None of that was a pre-registered hypothesis, and one failure in 48 is noise. It is consistent with the card’s explanation of its own gain, an adapter that finishes within the step budget, but it does not replicate it.
Why H5 has no verdict
My bench driver records pass status, run duration, and modified files, but not how many tool calls a run made, and each run overwrites its fixture directory, so there was nothing left to count afterwards. I did not guess. The card’s claim of less than half the turns is neither confirmed nor refuted here.
Why H6 failed
B matches C on passes and needs 3.4 times C’s minutes per pass. That is the price of the model, not the adapter. Qwen3.6-35B-A3B is a mixture of experts with about 3 billion active parameters per token, while the 27B is dense. On a box where decode speed is bound by memory bandwidth, the dense model reads far more weights per token.
A fault in my own instrument
The first attempt at arm C wrote 48 results of 0.0 minutes with “No provider configured”. My driver built the goose lane without the endpoint of the production server, so goose had no model to talk to. That is a fault in the instrument, not a result. I fixed it, moved the 48 non-runs into a separate file, and re-ran C in full.
Caveats
- A passing 47 of 48 runs created a ceiling that left no room to measure an improvement. The makers evaluated the adapter on tasks where their base model solves about 40 %, which is where an adapter has room to show what it does.
- The evaluation budget had no step cap. The card enforced an execution budget of 100 agent steps, and a substantial part of its gain occurred because the base model ran out of steps. My evaluation allowed 1,500 seconds per task with no step cap, so a step limit never cut the base model off.
- The evaluation was conducted on a single hardware configuration using a single software build. The tests ran on a GB10 chip using a community nightly build of vLLM without tuned LoRA kernel configurations for this GPU. The server logs recorded ‘Using default LoRA kernel configs’. The results here represent this specific build.
- These results are not comparable with the October benchmark figures. In the 192-run bake-off, pi running Qwen3.6-35B without a shell tool passed 28 of 48 runs. That figure reflected a different model, an older version of pi, no shell tool, and two broken tasks. It cannot be set against the 47 of 48 passes recorded here.
What I am keeping
The adapter remains a documented option rather than an active fleet lane. For my fleet, goose running on the production model is three times faster per correct result and does not need the production model paused.
What I learned is specific and useful. The LoRA loads and operates correctly on a 4-bit base that combines FP8 and NVFP4, costing 9 % decode speed. The makers’ warning about low-bit quantization drift did not manifest as a broken adapter or a failed task. At the same time, my 16 tasks have reached their ceiling. They still distinguish between harnesses, but they no longer separate a competent base model from an adapted one.
An experiment capable of measuring a difference would require harder tasks, the makers’ 100-step budget, and tool-call tracking on every run. That would be a separate evaluation with its own hypotheses, if the question warrants another evening of GPU time.
Sources
- The adapter and its card, revision
80d05c9: harness-hub/Qwen3.8-27B-pi-adapted; its GGUF variant - The base models: Qwen/Qwen3.8-27B and nvidia/Qwen3.8-27B-NVFP4 (its
hf_quant_config.jsonfor the per-layer formats) - The harness: pi; the bar: goose
- Training: TRL, AsyncGRPO, Harbor, OpenEnv
- The makers’ benchmarks: SWE-bench Pro hard 51, Terminal-Bench 2.1
- My earlier tests: the 192-run harness bake-off, Qwen3.8-27B against Qwen3.6-35B, Flash-Next on one Spark