Qwen3.8-Flash-Next Fits on One DGX Spark and Runs at 65 tok/s. It Still Did Not Earn the Slot.
My DGX Spark runs two local models in two roles. Role 1, the daily driver, is Qwen3.6-35B-A3B: fast, always on, good at the short tool-heavy turns my coding agents send it. Role 2 is for hard problems and runs on demand: since September that has been Qwen3.8-27B, which won my 28-task A/B against the 35B and then a four-model lineup test.
In daily use the 27B has not been the step up those scores promised. That is an impression, not a measurement, but it is why I looked again at the model I had ruled out in September: Qwen3.8-Flash-Next, the strongest Qwen that fits on one Spark at all.
This post is written in an unusual order. Everything above the results section was written before the model answered its first request: the sources, what I expect, the pass mark for each expectation, and the rule that decides whether Flash-Next replaces the 27B. The results are appended afterwards, and the text above them is not edited to fit.
Why I ruled it out in September
The September post cites a single-node comparison by Classmethod.
On one Spark they measured Flash-Next at 28 tok/s with 79 GB resident, a 14-minute start and nine community vLLM
patches. One finding mattered more than the speed: with tool_choice=required the model returned zero tool
calls. My agents live on tool calls, so that settled it.
What changed in six weeks
A community recipe by salient-data, announced in
a forum thread about six weeks of optimizing,
serves NVIDIA’s Qwen3.8-Flash-Next-NVFP4 checkpoint on
one Spark. What it publishes for its build of 2026-10-22, each row with its harness and raw data:
| Classmethod (September) | salient-data recipe (build 2026-10-22) | |
|---|---|---|
| Single-stream decode | 28 tok/s | 69.3 tok/s median over 40 prompts, prose 41.0 |
| Six parallel streams | not reported | 91.9 tok/s aggregate (one run) |
| Memory | 79 GB resident | 73.57 GiB weights after load, KV cache of 1.0 to 1.09 M tokens |
| Start to first answer | 14 minutes | 350 to 371 s on three boots |
| vLLM patches | 9 | 14, each with source and full diff |
| Quality check | not reported | GSM8K 99/100 on a fixed 100-problem subset |
The speed numbers come from different harnesses (theirs is bench_categories.py from another community
member), so I do not compare them with my own rulers directly. That is what the test below is for.
The recipe is also built to be checked. It pins the vLLM nightly by digest and the checkpoint to one exact commit, and ships the serving image signed with a cosign key that is in the repository. Its README says plainly that GB10 is outside the model card’s support statement, which lists B200 and B300 only.
What the public benchmarks say
On Artificial Analysis, Intelligence Index v5.0.1:
| Flash-Next | Qwen3.8-27B | Qwen3.6-35B-A3B | |
|---|---|---|---|
| Intelligence Index v5.0.1 | 29 | 20 (xhigh), 16 (low) | |
| Terminal-Bench 4.0 | 25.3 % | 2.5 % (low) | 0.0 % |
Sources: Flash-Next against the 27B, Flash-Next against the 35B.
Terminal-Bench is the row I care about most. It measures multi-step work in a shell, which is close to what my coding agents do, and there the gap is a factor of ten.
A correction on my own September post: it quotes the 27B at 34 and the 35B at 18 on the same index. Those numbers were from an older index version. Artificial Analysis restates the index from time to time, and numbers from two versions cannot be compared. Every row in the table above is from v5.0.1.
The license
The checkpoint is under the NVIDIA Open Model License plus the base model’s Qwen license. For a model that serves my own agents there is no restriction. The Qwen terms do ask for a separate license for offering the model as a service or building a hosted coding or office assistant on it, and above 100 million monthly users or 20 million US dollars of monthly revenue. If I ever expose it publicly, I read that again first.
Why a model larger than the machine fits
The checkpoint is 123.6 GiB on disk and the Spark has 121 GiB of memory, shared by CPU and GPU. No offload flag helps on that hardware, because “host” and “GPU” memory are the same pool. The recipe fits because its largest piece, a 51.2 GB n-gram embedding table, is never loaded. A patch reads only the rows each decode step needs, straight from the files on disk, through the page cache. What stays resident is about 74 GiB of weights plus the KV cache.
That still needs about 100 GiB free when the server starts. My production model alone takes about 46 GB, and the rest of the box (a web UI, a search engine, a relay, a CI runner, a few small apps) about 11 GiB. So the two cannot run side by side, and the test needs the box nearly empty.
Making room: a light mode
I wrote a small switch that empties the box and puts everything back. It stops the production model and every service that is not needed for an hour, pauses the timers that would start heavy work, writes down exactly what it stopped, and drops the page cache. Switching back starts exactly that list again and then the production model. Git, alerting, the local registry and the agent fleet’s coordinator keep running.
I also considered running headless, with the desktop session stopped, which would free another few GiB of browser memory. I left it out of the script: stopping the desktop also kills every terminal in it, including the one the agent driving this test runs in. The recipe needs about 100 GiB free; with production and the services stopped there is enough without it. If the load fails for lack of memory, headless from a phone is the fallback.
Getting it onto the box
- I read every script of the recipe I was going to run before running any: what it binds (localhost only), what it mounts, that it caps the container at 112 GiB and never asks for root.
- My Docker daemon cannot pull from GitHub’s registry directly, so the image went into a local registry with
skopeo, digest preserved. Before that,cosign verifyagainst the recipe’s public key. - The weights were downloaded at the exact revision the recipe was measured with, about 133 GB at roughly 9 MB/s on my line, with a watchdog that restarts stalled downloads. The recipe’s hash file for the small files (config, tokenizer) is checked before the first start.
- My launcher wraps the recipe’s own launch script instead of copying its flags, and refuses to start if the image is not byte-for-byte the build the recipe measured.
The download took most of a day, and a third of that was my own tooling. It started in the morning at about 9 MB/s. The last file, the 51 GB table, is too large for plain HTTPS and only comes through Hugging Face’s XET transfer, which my download script had switched off because it used to be flaky. The script retried the same error 180 times over an hour and a half before I noticed. It now switches XET on by itself for such a file and stops after three instant failures. The final run finished in the afternoon: 124 GiB on disk, no partial files, and all eight small files matching the recipe’s hashes.
The hypotheses, written before the run
Every check runs unattended in one script. It refuses to start while any agent task is running, switches to light mode, starts Flash-Next, runs the checks below in order, and on every exit, including a crash, stops Flash-Next and brings production back.
| Expectation | Source | Pass mark | |
|---|---|---|---|
| H1 | It loads and fits on this box, with the light mode | recipe: 350 to 371 s, KV 1.0 to 1.09 M tokens | answers within 10 minutes, KV cache at least 1.0 M tokens |
| H2 | The answers are correct, not just fluent | recipe: a loader that skips the n-gram table’s scale writes fluent wrong text | the recipe’s two known-answer checks pass |
| H3 | It is clearly faster than the 27B in role 2 | recipe: prose 41.0 tok/s; my 27B: prose 20 to 21 | prose at least 33 tok/s, code at least 55, on my usage-counted ruler |
| H4 | Forced tool calls work | Classmethod: zero calls with tool_choice=required | 6 of 6 correct single calls, three with thinking off, three on |
| H5 | It solves my 28-task set at least as well as the 27B | the 27B scored 25 of 28 on this box | at least 25 of 28 |
| H6 | The box comes back on its own | my light mode switch | production answers again after the run, every stopped service is back, no manual step |
Notes on the pass marks:
- H3 uses my own ruler, which counts the tokens the server reports, not stream events, so it will not reproduce the recipe’s 69.3, which is a median over 40 mixed prompts with another harness. The pass mark sits 20 percent under the recipe’s prose figure, and well above the 27B.
- H4 is new. My 28-task set only ever sends the default
tool_choice, so it could not have seen the failure Classmethod reported. As a baseline I ran the same check against my production 35B the same day: 4 of 4 correct calls. - H5 cannot show what I actually hope for. My set saturates: in the lineup test a much weaker model scored 28 of 28. With the 27B at 25, Flash-Next can win by at most three tasks, and one or two is noise. What H5 can show is a regression, plus two numbers I record without a pass mark: tokens per solved task and wall time for the whole set. All settings match the 27B’s row: temperature 0.6, top-p 0.95, budgets of 16,000 tokens for math and 12,000 for code, thinking on. The recipe turns thinking off by default, so the test turns it on per request; otherwise the comparison would be unfair to Flash-Next.
The decision rule, also written before the run
Flash-Next replaces the 27B in role 2 if H1, H2, H3, H4 and H5 all pass. The reason it may win on “no regression” instead of “measurably better”: its lead is attested by public benchmarks that are far larger than my 28 tasks and that I cannot reproduce on one box. My test has to show that the lead survives on this hardware, with this recipe, without breaking anything my agents depend on.
If any of them fails, the 27B stays, the post says which one and why, and the Flash-Next weights are deleted.
H6 does not decide the model, but if it fails, the light mode switch gets fixed before anything else runs on it.
If Flash-Next passes, a second, harder round may follow, because the 28-task set cannot show a lead. That round would get its own hypotheses, written before it runs, and would not change this decision after the fact.
Caveats, in advance
- One run per check, one sample per task. A one or two task difference on the 28-task set is noise.
- The recipe runs a vLLM nightly with 14 patches, outside the model card’s support statement. A result here is a result for this build, not for the model in general.
- Role 2 runs on demand. While Flash-Next is up, production is down, so my agents fall back to hosted models for that time. That cost exists with the 27B too.
Results
One unattended run on 2026-11-13. Every number is copied from the run’s own files.
| Pass mark | Measured | Verdict | |
|---|---|---|---|
| H1 | answers within 10 minutes, KV cache at least 1.0 M tokens | up after 382 s; 73.57 GiB of weights loaded in 144.9 s; KV cache 1,054,274 tokens | pass |
| H2 | the recipe’s two known-answer checks | 17 x 23 = 391 and Canberra, both right | pass |
| H3 | prose at least 33, code at least 55 tok/s | thinking off: prose 39.0, code 65.2. Thinking on: prose 38.5, code 39.5 | split |
| H4 | 6 of 6 forced tool calls | 6 of 6, three with thinking off and three with it on | pass |
| H5 | at least 25 of 28 | 23 of 28 (math 12 of 14, code 7 of 10, tools 4 of 4) | fail |
| H6 | the box comes back without a manual step | every stopped service and timer restarted, production answered again | pass |
The decision, by the rule above: H5 failed, so Qwen3.8-27B keeps role 2.
Two notes on the verdicts. My H3 pass mark did not say which thinking mode it meant. With thinking off the model clears both marks. With thinking on, which is how a hard-problem model is used, code drops to 39.5 tok/s, under the mark of 55 and still above the 27B’s 22 to 30. The speculative decoder accepts 4.4 tokens per step on plain code and 2.3 once the model reasons first. I call that a split, not a pass. It does not change the decision.
With the model loaded, the box had 13 GiB of memory left, and at the lowest point of the load 11.5 GiB. The light mode was necessary, not a precaution.
Where H5 failed
Flash-Next missed five tasks: two in math and three in code. Four of the five were not wrong answers. The model spent its whole budget, 16,000 tokens on math and 12,000 on code, inside its reasoning and never wrote the answer or the code block. One was a real mistake, a wrong count on the coins problem after 5,057 tokens.
The 27B missed three tasks under the same settings, two of them the same way. So both models overthink on this set. Flash-Next does it more often.
The same thing happened to my production 35B in September, and the cause is in the checkpoints. Both Qwen3.8 chat templates set the reasoning effort to xhigh whenever thinking is on and the request names no effort. I checked the 27B’s template after the run: it has the same default, so the comparison was fair. It also means both rows were measured at the most expensive setting, and neither at the one I would pick for daily use.
Two more runs, after the decision
These two runs happened after the rule had already decided. They are exploration and do not reopen it.
- A larger code budget. The three code tasks that ran out of tokens, repeated with 32,000 instead of 12,000: one passed after 4,982 tokens, the other two used all 32,000 and again wrote no code. Counting that pass, the score is 24 of 28, still under the 27B.
- The vendor’s sampling. The model card recommends temperature 1.0 and top-k 20 for thinking. With those, the full set scored 21 of 28, and all seven misses were budget cut-offs. The shared 0.6 preset was not the problem.
A lead worth following, not followed here
NVIDIA publishes a second checkpoint of this model, validated on the DGX Spark: Qwen3.8-Flash-Next-FP4-Local. Its card says it was made with quantization-aware distillation and “separate losses for end-of-response and end-of-thinking behavior”. The checkpoint I tested was quantized after training, for data-center GPUs. A model that cannot stop thinking, and a sibling trained specifically to stop, is a plausible pair. I have not tested the sibling. It needs a different serving stack that is still moving fast, and that is a separate experiment with its own hypotheses.
Caveats
- One run, one sample per task. I wrote before the run that a difference of one or two tasks on this set is noise, and 23 against 25 is inside that band. The rule was a threshold and I apply it as written. What the result supports is “not better on this set”, not “worse”.
- The set cannot show what the public benchmarks show. Their lead for Flash-Next is in multi-step agentic work. My 28 tasks are single hard questions, and a much weaker model once scored 28 of 28 on them.
- Both models ran at reasoning effort
xhigh, the templates’ default. A run atmediumfor both might change both rows. - The recipe is a vLLM nightly with 14 patches, outside the model card’s support statement. This is a result for this build on this box.
What I am keeping
The 27B stays the hard-problem model. Beyond the score, the two differ in what they cost to run. The 27B needs production paused. Flash-Next needs the whole machine: every other service stopped, six minutes to load, 13 GiB left. For an on-demand role that is a lot to pay for no measured gain.
The recipe did everything it promised. It fit a 124 GiB checkpoint into a 121 GiB machine, loaded in the time it said, answered correctly, and made tool calls that a September build could not. If you want Flash-Next on one Spark, it is the way to do it.
The rule says the weights go. I kept them one more day for a different question, whether a bigger local model writes better podcast dialogue than my daily driver. That test is in the script-writer post.
Three things outlast the scoreboard: a light mode that empties the box and puts it back, a download script that no longer retries a hopeless error 180 times, and a forced-tool-call check that my test set was missing.