EAGLE Throughput Is Content-Dependent: Same Run, 14 to 31 Tokens Per Second

I have been chasing podcast-script quality on my self-hosted stack and ran into a measurement that does not match the usual mental model of “tokens per second is a hardware property.” Same Mistral-Small-4 119B NVFP4 in the same SGLang container with the same EAGLE speculative-decoding config. Same GB10 box. Inside one pipeline run, throughput went from 13 to 31 tokens per second and back. Nothing on the hardware side changed. The only thing that changed was the kind of text the model was generating.

This post documents what I saw, the live SGLang logs that prove it, and the practical takeaway for anyone running long-context speculative decoding on consumer hardware.

The setup

The pipeline generates a 45-minute podcast script in two phases:

1. Initial generation. Two long-context calls to Mistral, each with a heavy system prompt that includes a numerical “output contract”: exact word-count targets, host word-share ratios, allowed turn-count ranges, hedge-opener minimums, mandatory cameo-segment counts.
2. Naturalize. A second pass that rewrites short batches of turns to add casual hedges and contractions. The system prompt for this pass is short. The user message just says “rewrite these turns to sound more spoken.”

Same SGLang container, same model, same GPU memory state. The only difference between phase one and phase two is the shape of the input.

What I expected

Mistral-Small-4 with EAGLE speculative decoding has historically run at about 25 to 35 tokens per second on this hardware. EAGLE accept rates of 0.7 to 0.9 are normal. I was expecting both phases to land somewhere in that range, with the naturalize phase a bit slower if anything because of the round-trip overhead from many small requests.

What I measured

SGLang’s scheduler logs every decode batch with its own throughput and EAGLE accept-rate stats. The container exposes them via docker logs:

[2026-05-06 18:36:11] Decode batch, gen throughput (token/s): 33.76, accept rate: 0.91
[2026-05-06 18:36:23] Decode batch, gen throughput (token/s): 29.04, accept rate: 0.72

I tailed those during the run and aggregated them in 60-second windows. Here is what the actual data looked like:

Same model, same GPU, same hour. Throughput more than doubled when the input changed.

Why EAGLE drops tokens on constrained output

This is the question that unlocks the diagnosis. EAGLE generates draft tokens with a small speculative model and verifies them with the full model. If the draft predicts the big model’s next token correctly, that token is accepted “for free.” That is why an accept rate of 0.9 is so much faster than 0.6: the verify overhead is amortized across more accepted tokens.

The draft model was trained on Mistral’s natural output distribution, which is why it predicts free-form prose well. When you inject a heavy structural contract into the system prompt, the main model starts steering toward tokens that satisfy the constraint set (schema, word counts, balance ratios). Those tokens are less “Mistral-typical,” which is why the draft model mispredicts them more often. Lower accept rate, more solo decodes, lower throughput.

Note: this mechanism is EAGLE-specific. Vanilla autoregressive decoding without speculative steps does not have an accept rate to tank, so you will not see this bifurcation on models running without EAGLE. The gap only appears when the draft model and the main model start diverging on what comes next.

The puzzle

If hardware was the bottleneck, throughput would not vary by a factor of two in the same run. If model size was the bottleneck, ditto. Memory bandwidth was at 96 percent utilization the whole time, so the GPU was working flat out in both phases. So what changed?

The answer is in the EAGLE accept rate. With three speculative steps and four draft tokens per step, an accept rate of 0.9 means roughly 3.6 tokens accepted per verify cycle. At 0.6 it is 2.4. The ratio of those two is roughly the ratio of throughputs I observed. The hardware had nothing to do with it.

The diagnosis

The initial-generation phase was producing text under heavy structural constraints: hit exactly six thousand seven hundred fifty words, balance host word-share to fifty percent, include exactly one VIBE cameo and one CLAWI cameo, ensure twenty-five hedge-openers across the episode, output as a JSON array with specific schema. All that pressure shows up in the actual tokens. The model generates with one eye on the constraint set, and that produces less typical Mistral output. The draft model has not seen that distribution at training time and starts mispredicting.

Caveat: the constraint set I described is unusually large. A prompt that enforces one or two rules (say, “output valid JSON” only) will not drop the accept rate as far. The effect scales with how many global constraints the model has to satisfy simultaneously. A single-field JSON output contract tested as of 2026-05-06 only knocked the accept rate from 0.88 to 0.81: noticeable, but not the 35-point drop the full podcast contract caused.

The naturalize phase is the opposite. The system prompt is short. The instruction is “rewrite these turns to sound more spoken.” There is no global word-count, no JSON schema, no balance ratio. Mistral writes the way it normally writes, and the draft model predicts well.

The accept-rate climb was visible in real time. During the Part 2 retry, where the prompt switches to “deepen and elaborate” rather than “satisfy these specs,” the accept rate climbed from 0.55 to 0.71 in the same generation. Loosen the constraint, get the speed back.

The trade-off

Slower throughput is not automatically worse. Here is the wall-time math.

Without the output contract, my v2 run produced four thousand four hundred fifty-three words against a target of six thousand seven hundred fifty. That is sixty-six percent of target, way under the floor, requiring a retry round-trip that adds two to three minutes.

With the output contract, my v3 run produced three thousand sixty-four words on Part 1 first try, ninety-one percent of target. Part 2 needed one retry to hit ninety-five percent. Total wall-time was about the same. The fast version that misses the target costs more in retries than the slow version that hits it on first try.

So the contract is worth it. But you have to know what you are buying.

This does not apply when your pipeline is latency-bound rather than quality-bound. If you are serving interactive responses and a user is waiting for each token, trading accept rate for precision is the wrong call. The wall-time math only works in batch-generation contexts where first-try success avoids an expensive round-trip. In a streaming chat context, a 14 tok/s decode feels sluggish in a way that 14 tok/s in a background script never does.

What this means in practice

Three things to take from this if you are running speculative decoding on your own hardware:

One, your throughput number is not a constant. Quote it as a range tied to the type of generation you are doing. “30 tokens per second on free-form prose, 14 tokens per second on structured output” is honest. “30 tokens per second” hides half the picture.

Two, EAGLE accept rate is a load-bearing metric. If your throughput tanks and your accept rate drops with it, the problem is your prompt forcing the model out of its draft-friendly distribution, not your hardware. Look at the prompt, not the GPU.

Three, structured contracts are not free, but the wall-time math can still work out. If a stricter prompt makes the output land on first try, the perf cost is paid back through fewer retries. Measure end-to-end, not just per-token.

The full live-monitoring pattern that exposed this is one line: docker logs sglang-mistral4 | grep "throughput". Worth tailing during long runs to see what your actual content does to your actual hardware. As of SGLang v0.4, these per-batch lines are written at every scheduler tick, so a 60-second tail gives you a real-time picture of how your prompt shapes acceptance.

What I am still figuring out

A few questions this raised that I do not have clean answers for yet:

• Whether you can fine-tune the EAGLE draft model on your own structured-output distribution and recover the speed without giving up the contract. This would matter for production agentic pipelines that emit JSON or function calls all the time.
• Whether mid-prompt content type predicts accept rate well enough to dispatch to a different inference path automatically, because that would enable a routing layer that picks speculative vs. standard decoding per request.
• Whether the same effect shows up on Mixture-of-Experts routing, or only on EAGLE. MoE expert selection is also distribution-dependent, so there is a plausible hypothesis that heavy constraints could push activation toward underfit experts.

Warning: the numbers in this article (v0.4, 119B parameters, GB10 hardware) reflect my setup as of May 2026. The accept-rate behavior is a structural property of speculative decoding and is not hardware-specific, but the absolute tok/s figures will differ on other memory configurations or future SGLang releases.

For now the practical move is the boring one: when speed matters more than format, prompt loosely. When format matters more than speed, prompt strictly. And measure both.