Three Local Models and Claude Wrote the Same Podcast Script. Bigger Was Not Better.
Hours earlier, a 180B model had failed my hard-problem test by two tasks out of 28, on a test set that could never have shown it winning. That is an unsatisfying way to learn something. So I asked a question where I already knew what bad looks like.
Vozonda is my open-source tool that turns an article into a two-host podcast episode. The script is written by the local model on my DGX Spark, Qwen3.6-35B-A3B, and its dialogue has had the same faults since September: every turn about the same length, almost no quick reactions, too many questions, and German that reads like a telegram. I work around that in code, with a repair pass and a rhythm layer that reshapes the script after the model wrote it.
The question: does a bigger local model write the dialogue better, so that the workaround has less to do?
The setup
No synthetic task. The harness builds the exact prompt Vozonda sends in production, captures it, and sends the same text to every candidate with Vozonda’s own settings: temperature 0.7, a 16,384-token limit, thinking off. It scores the raw script, before any repair or rhythm layer touches it.
- Two sources: the German Wikipedia article on podcasts and the English one on Nostr, each as an 8-minute episode.
- Three runs per source and model, because one script says little.
- Candidates: Qwen3.6-35B-A3B (production), Qwen3.8-27B, Qwen3.8-Flash-Next (all NVFP4 on vLLM, on the same box), and Claude Sonnet 5.5 as the hosted reference.
- A second pass with thinking on for the two Qwen3.8 models, two runs per source.
The score measures form, not whether the episode is good:
- Word budget: words written against words asked for.
- Quick reactions: the share of turns with five words or fewer (“Really?”, “Exactly.”).
- Questions per 1,000 words. NotebookLM’s episodes sit at 4 to 8 in my earlier comparison.
- Contractions per 1,000 words, the cheapest sign of spoken language.
- Rhythm problems: how many of the product’s own rhythm rules the raw script breaks.
Results
Means of three runs, range in brackets where it matters.
| Model | Source | Word budget | Quick reactions | Questions / 1,000 | Contractions / 1,000 | Rhythm problems | Seconds |
|---|---|---|---|---|---|---|---|
| Qwen3.6-35B (production) | DE | 89 % | 1 % | 20.7 | 0 | 1 | 23 |
| Qwen3.6-35B (production) | EN | 119 % | 0 % | 12.9 | 11.9 (0.8 to 31.6) | 1 | 23 |
| Qwen3.8-27B | DE | 93 % (64 to 111) | 16 % (5 to 29) | 12.6 | 0 | 2.3 | 89 |
| Qwen3.8-27B | EN | 83 % (55 to 109) | 7 % (0 to 16) | 9.8 | 0.9 | 2 | 59 |
| Qwen3.8-Flash-Next | DE | 54 % | 40 % | 12.5 | 0 | 2.7 | 29 |
| Qwen3.8-Flash-Next | EN | 60 % | 47 % | 16.2 | 4.1 | 3 | 26 |
| Claude Sonnet 5.5 | DE | 92 % (82 to 99) | 16 % (12 to 20) | 13.4 | 0 | 0 | 90 |
| Claude Sonnet 5.5 | EN | 83 % (78 to 91) | 15 % (10 to 19) | 10.2 | 25.1 (22.7 to 28.1) | 1.3 | 52 |
With thinking on, Qwen3.8-27B and Flash-Next returned no script in 8 of 8 runs. Each spent all 16,384 tokens reasoning, about five minutes on Flash-Next and ten on the 27B, and never started the JSON.
What the scripts look like
The same moment in the English episode, each from the first run, copied from the output files.
The 35B asks and answers in full sentences of the same weight, every time:
A: And users are identified by public keys, tagged as npub keys. How did this actually start? B: It was first written in 2020 by a Brazilian developer known as fiatjaf. He created it because he had disagreements with other protocols like ActivityPub and Secure Scuttlebutt, and he was frustrated with moderation issues on Twitter.
The 27B lets one host react before the next question:
A: Interesting. How does the data actually work? B: Data is stored in JSON blobs called Events, which are the only objects on the protocol. Users are identified by public keys tagged as npub.
Flash-Next reacts, and often does nothing else:
A: Who built it? B: A Brazilian developer using the pseudonym fiatjaf created it in 2020. He wanted an alternative to Twitter and had technical disagreements with protocols like ActivityPub and Secure Scuttlebutt. A: Interesting.
Sonnet is the only one whose host sounds like a person who wants something:
A: Protocol, not an app? B: Exactly, and that’s the key idea. Nostr can power social media, but also other kinds of applications. A: Okay, but how does the data actually work under the hood? Explain it like I’m new to this.
What I read from it
The 27B fixes the main fault. Quick reactions go from 1 percent to 16 in German and from none to 7 in English, and questions drop from 21 to 13 per 1,000 words in German. On the German source its form numbers are close to Sonnet’s on every column. It is also less steady: its word budget ranged from 55 to 111 percent across six runs, where Sonnet stayed between 78 and 99.
Flash-Next overcorrects. Nearly half its turns are quick reactions, and it delivers a bit more than half the words asked for. An 8-minute episode becomes a 4 to 5 minute one made of short exchanges. It is the fastest of the three new candidates, and that is the only column it wins.
Sonnet’s lead is in English, and in consistency. 25 contractions per 1,000 words against 1 for the 27B. It also broke none of the rhythm rules in German. On the counts in this table, the gap between the best local model and Sonnet is smaller than the gap between my production model and the 27B. Counts do not hear, so I would not stretch that sentence further.
Nobody wrote colloquial German. Zero contractions in every German script, Sonnet included. When the strongest model in the test misses in the same way as the weakest, the fault is in my prompt, not in the models. That is the cheapest fix this test found.
Thinking mode is a trap for this job. Vozonda already switches thinking off for script calls, because the 35B once reasoned its way through the whole token budget. The Qwen3.8 models do the same thing more reliably: both templates default to the highest reasoning effort when thinking is on.
Why the 27B is not simply the new writer
Speed is not the reason. A script is written once per episode, and 60 to 90 seconds disappear next to the minutes the speech synthesis takes.
Memory is the reason. On this box two large models do not fit side by side. The 27B can only run while the production model is stopped, and the production model also serves my coding agents all day. Swapping models for every episode is possible for a batch at night and wrong for an app that should answer when someone presses a button.
Flash-Next has that problem at full size. It needs the entire machine, every other service stopped, for a result that is worse here than the 27B’s.
What follows
- The prompt gets fixed first. Spoken German forms, asked for explicitly and with the rule that examples are never copied. Every model in this test benefits, including the one that is already loaded.
- The 35B stays the default writer, with the rhythm layer doing what it does today. The layer exists because of the faults this table shows, and it is cheaper than a second model.
- A hosted writer is a real option for this one step. A script is a few thousand tokens. The heavy part of an episode, speech synthesis and mixing, stays local either way. Vozonda already has the switch for a cloud writer; what it lacks since late September is a provider that stays up.
- The 27B is the writer for batch runs when production can pause.
Caveats
- Form counts, not quality. Whether an episode is pleasant to listen to is decided by ear, and that listening test is still to do. I built a blind pack of four German scripts with the model names removed for it.
- Three runs per cell, two sources, both Wikipedia articles. The ranges show how much one run can differ.
- Sonnet ran through the Claude Code command line with tools off, not through the bare API, so a system prompt of its own preceded mine.
- I also ran Gemini 3.6 Flash and do not report it: its hidden reasoning used up the token limit and every script was cut off after about 650 tokens. That is my harness’s mistake, not a result.
- All local models are 4-bit NVFP4 builds. Full-precision versions may write differently.