I Upgraded vLLM Under a Live Server. My Own Watchdog Was the First Thing to Break.

My daily driver on the DGX Spark is Qwen3.6-35B-A3B in NVIDIA’s NVFP4 build, served by vLLM. The image under it was a community nightly with vLLM 0.23.1rc1. It worked, it matched the Spark Arena recipe flag for flag, and it had crashed once the day before with CUDA error: an illegal memory access on a code path that only runs with --async-scheduling. A newer nightly with vLLM 0.32 had been on disk for a day, pulled for a model I ended up not adopting.

So the question was simple: is the newer engine better for the model I actually run, and which of its knobs matter? The answer took two unattended rounds, one bug in my own tooling, and one benchmark result that looked like a win until the second round showed it was noise.

What I measured, and with which rulers

Every variant was measured with the same three tools I use for every model decision on this box:

• Decode speed, counted from the server’s own usage.completion_tokens, on prose and on code, with thinking on and off. Not stream events: with MTP speculative decoding one event carries two to four tokens, and a ruler that counts events reports a third of the real speed.
• Prefill speed, which I had never measured before this test. A prompt of about 8,000 and about 30,000 tokens, opened with a random nonce so the prefix cache can never serve it, max_tokens 1, three runs each after a warm-up. Coding agents send big repository context and get short answers back, so a decode-only benchmark misses most of what they wait for.
• Quality, on my 28-task set: 14 math and logic problems with computed answers, 10 coding tasks run against hidden tests in a container with no network, and 4 tool-calling checks. Temperature 0.6, one sample each, 32,000-token budgets.

The variants were relaunches of the same launcher script with environment overrides: QWEN_IMAGE, MOE_BACKEND, ASYNC_SCHED, MTP_K. The defaults stay production. A variant becomes production only when someone edits the defaults after a measured win, never because an environment variable was left set.

Running it without anyone watching

The Spark has one GPU and one big model at a time. Testing an engine means taking production down, and the agent fleet uses that model around the clock. The harness had to be safe enough to run while I was away from the desk:

• It waits until production has served no request for a few minutes before each relaunch, and gives up after four hours.
• A memory guard stops a candidate at once if free unified memory drops under 8 GB. On this machine that is where the desktop starts to freeze.
• An EXIT trap relaunches production with its defaults, whatever happens: a crash, a timeout, a guard stop, or a normal end.
• Each variant writes an info file with the vLLM version reported from inside the container, the image, the MTP acceptance from the log, and a count of CUDA errors.

That last file is the reason I trust round two. In round one it is how I found out that one of my results was measured on the wrong engine.

Round one, and the watchdog

Production has a watchdog: every five minutes it asks the model for one generated token. Two failures in a row and it restarts the engine and sends me an alert. It also has a boot grace, so a container younger than 15 minutes that does not answer yet is labelled “loading” instead of “hung”.

Round one ran four variants back to back. The variant without async scheduling was booting when the watchdog ran: strike one. Five minutes later the MTP k=2 variant was booting: strike two. The watchdog did exactly what it was built to do. It restarted production with the defaults, filed a forensic bundle for a hang, and sent me an alert that read like a real outage.

The bug was in the grace. It changed the label to “loading” but still counted the strike. Two different boots, five minutes apart, added up to one restart. In normal operation a boot happens once, so the counter never saw two in a row. The A/B harness was the first thing that booted the engine repeatedly, and it found the gap in a few hours. The fix is small: a young container that does not answer now resets the strike count and exits before the counter moves, with a test that replays the sequence.

The harness then measured “k=2” on whatever was running. The info file said vLLM 0.23. Without that one line I would have published a k=2 result for an engine that never ran it.

Round one still answered the main question:

Decode in tok/s, thinking off and on. The newer engine is faster on every workload. And the native NVFP4 kernel scored 26 of 28, the best result this model has ever had on my set.

Round two: the win that turned out to be noise

vLLM 0.32 ships a MoE backend called b12x, with native NVFP4 kernels for the Blackwell consumer and Spark chips. NVFP4 weights and NVFP4 activations (W4A4), where the Marlin backend I run dequantizes to 16-bit activations (W4A16). On paper the native path should be faster. Combined with the 26 of 28, it looked like the obvious pick.

Round two ran with async scheduling off for every variant, because round one showed it bought nothing measurable, and the crash the day before happened on its code path:

The native kernel dropped from 26 to 23. Same weights, same prompts, same budgets; the only difference from round one is async scheduling, which does not change what the model computes. A spread of three points between two runs of the same setup is what one sample at temperature 0.6 does on 28 tasks. I had written down a noise level of about two for this set. It is closer to three.

The native kernel also lost on prefill, about 10 percent at 8,000 tokens, and one of its prose runs fell to 9.8 tok/s against a median of 80. Native W4A4 is a real step for the hardware, but on this model and this image Marlin is still the faster and steadier choice. I will look again when the next image ships.

MTP depth behaved the way speculative decoding usually does. With k=2 the draft head proposes fewer tokens per step, and code, the most predictable text, loses the most: 126 against 135. With k=4 more drafts get rejected on prose, and the extra draft work costs more than it saves: 83 against 95. Three stays. The same content dependence showed up earlier with EAGLE: when speculative decoding helps and when it does not.

What changed in production

Production now runs vLLM 0.32 with Marlin, MTP k=3 and async scheduling off. Against the old engine, measured again after the switch:

The prefill line is the one my agents feel. A 30,000-token repository context used to take 5.8 seconds before the first token. It now takes 4.7. Every agent turn pays that again.

The quality line I am not claiming as a win. 25 against 22 is inside the noise round two just measured. What it shows is that the new engine is not worse, which is all an engine upgrade has to show.

Two problems before the first measurement

Two practical problems came before the first measurement.

The tag I pinned was gone. My launcher had pinned the image by tag. The publisher had since removed the v0.23.1rc1 tag from the registry; only dated nightly tags remain. The image itself was still there under its digest, so production would have kept starting, but a rebuild from scratch would have failed. Both launchers now pin by digest, with the tag as a comment, and a small read-only check reports for each launcher whether the tag still points at the pinned digest, has moved, or is gone.

The registry refuses my network path. All container traffic on this box goes out through Tor, and the GitHub container registry answers pulls over Tor with “denied”. I copied both images with skopeo copy --preserve-digests into a local registry:2, and the launcher now pulls from there by the same digest. The digest is identical to the upstream one, so the pin still means what it says, and the check asks the upstream registry about the tag. The side effect I like: rolling back to the old engine no longer depends on someone else’s registry. It is one environment variable away, from a copy on my own disk.

What I would tell someone doing the same

• Measure prefill. Decode gets all the attention, but the biggest gain of this upgrade was prefill, and I would have missed it with my old rulers.
• Run the quality set twice before you believe a three-point gap. My best result ever was noise.
• “Native” is not “faster”. The W4A4 kernel is the right direction for this hardware. On this model and this image it lost on prefill and on stability.
• Async scheduling is not free speed. Here it bought nothing measurable, and it is the prime suspect of a crash. It stays off unless a measurement says otherwise.
• Your safety net meets your test harness for the first time on the day you test. The watchdog was correct for every situation it had seen. A test that boots the engine four times in an hour was not one of them.
• Log the engine version from inside the container for every result. It is one line, and it caught a result measured on the wrong engine.

What is still open

Stability. The old engine had eight hangs and one crash in three weeks, most of them under agent load. Turning async scheduling off and moving to 0.32 removes the prime suspect, but a suspect is not a cause. I will count hangs and GPU errors for four days of normal agent load and compare. If 0.32 hangs, rolling back is one variable and a restart, and the old engine is on my own disk.