"125 billion parameters on one gaming GPU — first impressions of Strata + Qwen 3.8 Flash Next"

2026-10-27 · AI & Tech · local AI MoE GPUs inference quantisation · Source: Watch the review

In brief — 5 key takeaways:

  1. 125B on a 4090 at 140 tok/s: with Strata + Qwen3.8-Flash-Next (IQ3_S quant), a single RTX 4090, 64GB RAM and a Ryzen 9 7950X hit ~140 tokens/s decode, ~4,400 tok/s prefill and a full 256K context window — and people are reportedly running it on 12–16GB cards.
  2. MoE is why it's possible: only ~6B of the 125B parameters are active per token, so compute isn't the blocker; Strata solves the memory side by spreading weights across GPU VRAM (hot experts), system RAM, and SSD.
  3. The agent's benchmark was wrong — and the catch was real: the first reported 203 tok/s / 7,800 tok/s prefill ignored reasoning-token time and prefix-cache hits; honest repeatable numbers were ~140 / ~4,400, which are still exceptional for local hardware.
  4. Skip the manual tuning; keep KV grow: a huge hand-run parameter matrix (pcie-frac, CPU workers, MTP depth and confidence) found no real gain over Strata's automatic calibration — noise beat the tweaks. The one standout was KV grow, which reserves the KV cache lazily instead of upfront, keeping short-context fast while enabling 256K at only ~140→124 tok/s when the window is actually full.
  5. Six weeks from toy to tool: six weeks earlier the same model crawled at ~10 tok/s on llama.cpp with aggressive IQ1 quants; Strata's arrival a month later plus two weeks of community optimisation took it to genuinely work-capable — a real frontend task went through one-shot in about ten minutes.

The review

A ~20-minute first-impressions video from y8y.systems (posted 15 October 2026) on running Qwen3.8-Flash-Next — a 125-billion-parameter Mixture-of-Experts model — on one RTX 4090 using the open-source Strata inference engine. The premise: "125B parameters on a 24GB GPU? Local AI is getting ridiculous."

Why it fits and why it's fast

The obvious physics problem has two halves: can you fit the model, and can you actually compute it. The quantised files alone exceed 80GB — that's not fitting into 24GB of VRAM, "not even into 48GB." And running all 125B parameters per token on a gaming GPU would mean single-digit tokens per second.

Two things break the problem open:

  • MoE routing. Only about 6 billion parameters are active for each generated token. The reviewer's analogy: a dense model is a company where every department touches every task; MoE calls in just a small expert panel. He's careful to add that experts aren't literally "the math person" and "the programming person" — routing doesn't map to human job titles.
  • Three-tier memory in Strata. Hot (frequently used) experts are cached in GPU VRAM, the full expert weights live in system RAM, and big supporting structures sit on fast SSD. "The important question isn't how big your desk is — it's how efficiently you can get the right document when you need it."

His config runs the higher-precision IQ3_S quant (Strata recommends IQ2_XS for 64GB machines; he was "dancing right on the edge of an OOM error"). The whole setup uses 50GB+ of system RAM and essentially all 24GB of VRAM.

The benchmark that wasn't — and the honest numbers that still are

The agent's first benchmark reported 203 tok/s decode and 7,800 tok/s prefill. Too good to be true: it didn't account for reasoning-token generation time or prefix-cache hits. A proper repeatable benchmark suite produced ~140 tok/s decode and ~4,400 tok/s prefill on fresh input. The reviewer's caveat is fair — 4,400 only sounds disappointing next to fast cloud APIs like DeepSeek V4.1 Flash; for local hardware it's "genuinely, seriously good," and in some workflows it feels faster than online APIs.

He gave it a real job — hiding a table-of-contents sidebar behind a button in his own document-editing frontend — and it completed investigation, implementation, testing and verification in just over ten minutes through an agent framework, one-shot, before the config was even fully optimised.

Tuning findings worth stealing

  • Automatic calibration wins. Strata ships hardware auto-calibration. After a full interleaved A/B matrix of manual settings — the pcie-frac CPU/GPU work split, CPU worker counts, multi-token-prediction depth and confidence threshold — apparent 5–9% gains vanished on proper repeats, sometimes inverting. Run-to-run variance (which experts route, MTP acceptance, cache state) exceeded the effect size. "Unless you're genuinely enjoying the adrenaline rush of chasing that final 1%, just run the official calibration."
  • GPU power limit: 320W, not 450W. Max power was ~5% faster at short (32K) context but slower at long context, with far more heat. 320W was the sweet spot; 280W if heat and power matter more than long-context throughput.
  • 256K context against the recommended 128K — works. The model's per-token KV cache is small, so a 256K window fits in the VRAM budget; the cost is fewer hot experts cached, so decode at a full 256K window drops from ~140 to ~124 tok/s. "Losing a bit of speed in exchange for doubling the available context is a deal I'd happily take."
  • KV grow is the standout. Normally a 256K maximum reserves the whole KV cache upfront, so even a "hello" pays in decoding speed. KV grow expands the cache only as the conversation actually lengthens. Interleaved tests found no stability regressions (long-context retrieval, 30-turn agent sessions, tool-call success, repeated load/unload), and short-context decode and prefill got faster. Only when the window is completely full does it give back a tiny, within-noise amount of speed. "A really elegant design... I'm absolutely keeping it enabled."

The pace point that sticks

Six weeks before the video, the same model ran on 4090s at roughly 10 tok/s decode and sub-100 prefill using desperate IQ1 quants — "impressive that it ran at all," unusable for real work. A month later Strata appeared; two weeks of further optimisation later, it was doing production coding tasks. Model size and VRAM are "no longer a simple one-to-one relationship" — MoE, quantisation and tiered memory broke the equation, and the community's iteration speed is only accelerating.

A note on sources: this summary was built from the video's auto-captions, which mishear proper nouns and numbers. All figures are the reviewer's own measurements on his own hardware and are reported, not independently verified.