All articles tagged "benchmarking" : self-hosted AI fixes, setups, and architecture notes.
An unattended A/B of six engine variants for Qwen3.6-35B on one DGX Spark: vLLM 0.23 against 0.32, Marlin against a native NVFP4 MoE kernel, async scheduling on and off, three MTP depths. Prefill got 51 percent faster, the native kernel lost, async scheduling bought nothing, and the safety net killed a test run.
Read article →
GitHub's Project HydraFusion and HydraFusion Max route coding requests through Single, Cascade, and Critique patterns in the cloud. Sovereign Fleet runs those same three patterns unattended on your own Linux machine across Goose, Aider, Pi, and OpenCode inside a Landlock kernel sandbox.
I benchmarked goose, aider, Pi and opencode across 192 runs on a local Qwen3.6-35B. Aider passed 39 of 48 tasks at 3.0 minutes per correct result, while goose achieved 38 of 48 correct results at 1.6 minutes per correct result.
Aleph Alpha released Kolibri-1 under Apache 2.0: a 78B MoE with 3.5B active parameters and a native German tokenizer. On my DGX Spark I measured the tokenizer (15 percent fewer tokens on a real German prompt) and the memory math, and the server never got past startup at 64k context. Independent and vendor benchmarks show why it will not unseat Qwen on my hardware.
Autonomous coding agents can land dozens of tasks in a day, but passing tests only proves code executes, not that the underlying claim holds. Here is the operational reality of our review loops, benchmark traps, and the decision to freeze development.
I rebuilt the local model lineup on a single DGX Spark with the same two rulers for every candidate: a deterministic 28-task test and a real agent loop. Gemma-4-26B scored 28 of 28, a 2B model beat my 35B daily driver, and two big models at once crashed into a memory peak nobody models. What I kept, what I deleted, and why a perfect score was the wrong answer.
I checked my DGX Spark's Qwen3.6-35B against its Spark Arena recipe, found a benchmark tool that counts stream events instead of tokens, then ran a harder deterministic A/B against Qwen3.8-27B: 24 of 28 against 15, mostly because the 35B overthinks. Along the way the benchmark crashed production three times and exposed a watchdog bug.
My self-hosted coding-agent fleet failed task after task, on every model. The models were not the problem: seven plumbing bugs were, from a 200-character task cut to 430k tokens pushed into a 262k window. Here is each bug with its evidence and fix, plus a pytest-judged bake-off of aider, goose and opencode on a local Qwen and free NVIDIA NIM models.
Qwen3.6-35B built a working SvelteKit 5 + D3.js app from a single prompt. The build passes. The features ship. But the architecture is improvised, the code is duplicated, and it took eight debugging iterations to get here. Here's what the gap between 'works' and 'good' actually costs.
Gemma-4-31B in NVIDIA's NVFP4 format fits a single DGX Spark and is a strong reasoner. But on Blackwell sm_121 the default FP4 kernel path is broken, and a dense 31B is bandwidth-bound at around 4 tok/s no matter what you do. I measured the baseline, the Marlin fix, and the honest conclusion: the real speedup is a model swap, not a flag.
GLM-4.7-Flash is a 30B-A3B MoE coding model that fits a single 128GB DGX Spark with room to spare. Bringing it up on Blackwell sm_121 took two failures that every published recipe gets wrong: the 'AWQ' build is actually compressed-tensors, and the model speaks MLA, so flash_attn is illegal. Here is the working recipe, the single-stream decode number nobody reports, and what it does to my coding agent.
I almost bolted a vector database onto a knowledge base that did not need one. My own benchmark told me to, then told me the opposite, both times with total confidence, both times because I had quietly chosen the test queries. Here is the path from the standard 2026 RAG playbook to the number that finally told the truth, and why the retriever that won is a hundred lines of standard library.
Standing up two large models on a DGX Spark, my own measurements tried to deceive me three separate ways: a harness that scored a working model at zero, a one-shot test that framed the model for a bug that was mine, and a cold reading that undersold decode speed by 35 percent. None of the wrong numbers were random. Each had a cause, a tell, and a fix. Here is the field guide.
gpt-oss-120b pulls nearly four million downloads a month, so I assumed it was a one-command experience. Getting it to serve on a DGX Spark took a frozen box, a 25GB image pull strangled by a Tor proxy, and a 43-minute kernel compile. Then the measurement: on my own coding tasks the 120B scored 56 percent where the 35B Qwen I already run scored 100. Here is the full teardown, with every number measured on the box and the failed measurements thrown out, not published.
Same model, same box, three ways to shrink it: Intel's AutoRound int4, a 4.75-bit PrismaQuant, and FP8. I measured all three on decode speed, coding accuracy, and vision, with one ruler per axis and the failed runs thrown out. AutoRound won every column that mattered, and the surprise was vision: the leanest build kept its eyes while the others went blind or broke. Here is the teardown.
NVIDIA's Nemotron-3-Super-120B-A12B is tuned for Blackwell and ships an NVFP4 build that fits a single 128GB DGX Spark. I measured it where almost nobody else does: single-stream, on one GB10. The result is 23.7 tok/s, a competent but painfully verbose coder, and a genuinely strong retrieval agent. Here is the full teardown, with the published benchmarks fact-checked against what the box actually did.
I built a small, dependency-free harness that answers one question with numbers instead of vibes: does this enhancement make my agent measurably better, on my models, on my tasks? Here is the method, what I found, and why deterministic gates are the whole point.
I run Qwen3.6-35B at 4.75-bit for coding. A 4.0-bit AutoRound build promised more speed. Fewer bits usually means a dumber model, so I measured both halves: decode throughput and coding quality, the latter through my own agent-bench harness. The result settled it. Here is the duel, the bandwidth math, and why the bit count was the wrong thing to fear.
caveman has ~200k installs and claims 75% token reduction. I measured it on two local models and three Claude frontiers (Sonnet 4.6, Opus 4.8, Fable 5). The math does not work out the way the claim says it does.
Serena is one of the most-installed coding MCP servers. I tested it against two local models (Qwen3.6-35b and Mistral-Small-4) on three refactor tasks with deterministic gates. The short answer is more interesting than yes or no.