A Newer vLLM Nightly Gained Nothing on My DGX Spark. I Kept the Old One.
Quick take: I tested a three-week-newer vLLM nightly against the one in production. Decode and prefill moved by less than 4 percent, in both directions, and the newer build solved two tasks fewer. Production stays where it is. Further down: what else changed on the grid this week.
What pushed me to look at the 0.33 nightly
The engine under my daily model has not changed since the last unattended engine test: a community nightly from 2026-10-16 with vLLM 0.32.0rc1, pinned by digest, serving Qwen3.6-35B-A3B in the NVFP4 build. It has been quiet. The server counters at the start of this week showed a prefix cache hit rate of 89 percent, an MTP draft acceptance of 71 percent (83, 69 and 59 percent for the first, second and third drafted token), and zero preemptions, restarts or GPU errors.
The nightly from 2026-11-07 with vLLM 0.33.0rc1 sat there, and three weeks of engine commits can hide a real gain. And the top entry for this model on the Spark Arena leaderboard reports 178 tok/s. That number comes from a different benchmark tool and a different load pattern than mine, so I cannot put it next to my own figures. What I can do is check whether its recipe differs from mine. It does in one flag that matters: it runs with --async-scheduling. I switched that off on 2026-10-24 because it crashed this exact setup with an illegal memory access and bought no speed in my measurements. Same flags, new image: I kept async off for this test.
The test
The harness is the one from October, unchanged. It waited until production had served no request for ten minutes, measured production, relaunched the same launcher script with one environment override for the image, measured again, and then restored production from an exit trap. Start to finish it took 71 minutes, and for about half an hour of that the grid had a test engine instead of its normal one.
Three rulers, the same ones I use for every model and engine decision here:
- Decode speed from the server’s own
usage.completion_tokens, on prose and on code, with thinking on and off. - Prefill speed on prompts of about 7,600 and 30,400 tokens that the prefix cache cannot serve.
- Quality on 28 tasks: 14 math and logic problems, 10 coding tasks against hidden tests, 4 tool-calling checks, with a 32,768-token budget each.
| Measurement | Production (vLLM 0.32, nightly 2026-10-16) | Candidate (vLLM 0.33, nightly 2026-11-07) |
|---|---|---|
| Prose, thinking off | 98.5 tok/s | 95.1 tok/s |
| Prose, thinking on | 106.5 tok/s | 108.3 tok/s |
| Code, thinking off | 142.8 tok/s | 145.2 tok/s |
| Code, thinking on | 121.6 tok/s | 122.0 tok/s |
| Prefill, 7.6k tokens | 8,212 tok/s | 8,370 tok/s |
| Prefill, 30k tokens | 6,563 tok/s | 6,592 tok/s |
| Quality, 28 tasks | 24 passed | 22 passed |
| Benchmark wall time | 1,441 s | 1,507 s |
| CUDA errors in the log | 0 | 0 |
Reading it
The largest speed difference is prose without thinking, 3.4 percent in favour of the old image. The largest in the other direction is short prefill, 1.9 percent. Both sit inside the range of the individual runs: prose without thinking ranged from 92.3 to 101.9 tok/s on production alone. There is no speed signal in this table.
Quality needs more care. The candidate failed one more math task and one coding task. Every failure in both runs is a task that used its full 32,768-token budget without finishing, including the coding failure, which ended with no code block at all. With one sample per task at temperature 0.6, two tasks tipping over the budget is something I have seen between two runs of the same engine. So 0.33 is not better. That is enough to decide.
An upgrade with no measured gain is only risk. The candidate also needed 290 seconds from launch to ready, which is in line with the old image, so there is no startup argument either. Production keeps the 0.32 nightly, and the launcher defaults were not touched.
One thing this result does not answer is whether async scheduling is stable on 0.33. It is the only lever left that could explain the gap to the leaderboard recipe, and I need a run with the crash counter in view.
The rest of the week
sovereign-fleet 0.11.0: tasks that wait for each other
The release is on PyPI since today. The headline change is that depends_on now works on both execution backends, so a coordinator can dispatch a chain of tasks in one go and the second step starts only when the first one is completed. Before, a chain meant waking up after every step to dispatch the next.
It does not make the work itself faster. The agents run the same steps in the same time. It saves coordinator attention though, and with a cloud model as coordinator that is tokens. Review moves to the end of the chain so a wrong first step can carry through. I use chains for steps that build on each other mechanically, and I keep single dispatch for anything where I want to read the result first.
Using the fleet on real work this week surfaced three fixes that went into the same release:
- A retried task now starts from the current default branch. It used to reuse the base of its first run, so a fix that landed in between never reached the retry and it failed the same way again.
- When committing an agent’s work fails, the worktree stays. It used to be removed, and the work with it.
- An agent’s whole process group is ended on every exit path, and captured output is capped at 10 MiB per stream.
The fleet’s own score for the day: nine tasks dispatched, usable work from all of them, seven landed without my hands on git. The fleet’s reviewer caught one critical bug before landing and waved through two deviations from the task that I caught afterwards. A second reader is still required. The comparison with other orchestrators describes that review stage.
A dashboard that loads nothing from outside
The grid dashboard is the one page I open to see whether everything is healthy, so it should not depend on anything it monitors, or on the internet. It loaded React from a CDN. Those files are now vendored and served by the dashboard itself, which made a strict Content-Security-Policy possible. Cross-origin access is off unless an allowlist is set.
Every open tab used to trigger its own full status collection. Now all clients share one result for three seconds, and a hidden tab stops polling. The reverse proxy also compresses responses: the main page went from 188 KB to 59 KB over the wire.
Services that notice they run old code
My status page kept showing the same warning class: a service still running the code from before the last commit. Each one was a manual restart, and each restart had to wait until no task was running. A timer now does this every five minutes. It compares the start time of a service with the newest commit that touches its code, ignores commits that only change tests or review notes, and restarts only when the queues are idle. The warning now means something again: if it shows up, the automatic path failed.
82 GB back, and one script that gave up too early
A cleanup pass freed 82 GB on the data disk, about 32 GB of it container build cache, and removed 37 task branches that had long been merged. Models and the local registry were left alone. The registry rule from the mirror script holds here too: never garbage-collect with --delete-untagged when production images are pinned by digest.
The daily build of the Vozonda website failed once this week, one second after its timer fired, on a single git pull. The script aborted on the first network error and hid git’s message. Network steps now retry up to four times with a doubling pause, and the real error is logged.
One mistake of my own
While patching the status script, a one-line helper opened the file for writing and read from the same file in the same expression. The open truncated it first. The result was an empty file where a script of more than 1,200 lines had been, uncommitted changes from another agent included. Git had the committed part. The rest I rebuilt from that agent’s session log.
Write to a temporary file, then rename it over the target. A rename is atomic, a truncate is not.
Still open
- Async scheduling on vLLM 0.33, as its own run.
- The fleet’s router still cannot know in advance how much daily quota a lane has left. The damage from guessing wrong is contained now, the guess itself is not fixed.
- A quality set with more than one sample per task. I need to know whether a two-task gap is real or just noise.