Strata forks compared: Unbiased Bob's live benchmark stream — key findings

2026-10-26 · AI & Tech · local AI Strata benchmarks · Source: Watch the stream

Key findings — "Profiling Strata Forks: Maya vs. Unbiased vs. PolyStrata | LIVE"

Source: https://www.youtube.com/live/q3TC6-RDOjA — Unbiased Bob (4.39k subs), posted 2026-10-18, 2h40m livestream, ~5.5k views. Part three of his Strata series. Method: Full auto-caption transcript fetched 2026-10-26 (youtube-transcript-api), split into 4 chunks, each summarised in full by a separate reader pass, then merged. Transcript is speech-to-text: proper nouns and numbers are frequently garbled — labels below are quoted as heard; treat uncertain ones as flagged, not confirmed. Full timestamped transcript: 2026-10-25 YouTube transcript - q3TC6-RDOjA.txt (same folder).

What the stream actually was

A working session where Bob builds his own fork of Strata ("UnbiasedStrata", plus a second branch "unbiased strata two" targeting PolyStrata), and live-tests models on rented RunPod GPUs. Cast of engines: Base Strata, PolyStrata (rumoured-banned fork, "Strata with an additional engine tacked on"), llama.cpp mainline (now with MoE expert-cache merged), and his fork(s).

Strata praise (streamer claims)

  • Strata's MoE offloading ("optimized streamlined inference engine") is the big deal: llama.cpp/vLLM/SGLang "had bits and pieces," but "as far as I know, Strata and Qwen 3.8 Flash Next was the first time you could take something off the shelf and run 177 billion parameters on a 16 gig card."
  • Named features: VRAM expert cache, N-gram state on SSD, an optimisation pass, "like a planner."
  • "The strength of this Qwen 3.8 Flash Next model is insane… very, very strong."

Strata criticisms (directly relevant to our setup)

  1. Install method = supply-chain risk. Recommended install is "grab the markdown and eat that into your agent" (docs/AI_setup). A bad commit could install anything — "your threat model is now the latest commit on master of Strata." Contrast llama.cpp: clone, build with CMake.
  2. Runtime auto-updates: Strata downloads binaries from GitHub and overwrites what's running locally — unacceptable "for someone in the business of benchmarking" who needs provenance.
  3. He says maintainer "Nico" acknowledged both issues on Discord and said he'd fix them — "and he hasn't done anything about it." Bob is fair about it (no obligation; he doesn't envy the maintainer position) and concedes commenters are "kind of right" that there's no evidence Strata is actually doing anything wrong.

Measured benchmarks reported (his own runs, exact labels preserved)

  • RunPod "cheap ones, 3090s with 100 gigs of RAM, ~50c/hr": PolyStrata and Base Strata ~120 tok/s on Qwen 3.8 Flash Next; llama.cpp "much less, 70 or 80"; llama.cpp "five times slower on prompt processing" — "severely behind." Purpose: validate PolyStrata is real.
  • Yesterday's baselines (3090): both PolyStrata and Strata ~112 tok/s decode, ~2400 prompt processing on Flash Next.
  • Bob Bench Wide, 16 GB VRAM + 32 GB RAM envelope (his 4090): Strata Coder variant of Qwen 3.8 Flash Next ("super quantized, and then they deleted a bunch of experts") scored 89.9 / ~90% — same as Qwen 35B-A3B → "that's bad news." Chat note: IQ3XXS got 93%. Run details: passed "BBW Alpha audit," <32 GB system RAM, avg 105 tok/s, draft acceptance ~0.698. Config caveat: Coder ran "Q4 KB cache at 128k context" vs the IQ3 run at higher context — not apples-to-apples. "I just wasted 6 hours of my own GPU."
  • Pareto curve he maintains: BBW score vs time-to-run. Current best for 16 GB cards (5060/5060 Ti/5080, fully resident): dense 27B Qwen 3.8. New point: "90 in 6 hours."
  • DeepSeek "4.1 Flash 1017" (dropped the day before): ~"552 B" total, activates 8 B prefill / 16 B decode; same architecture as before, weights-only change. On 2× B200 RunPod (183 GB cards): DS4 at ~50 tok/s, using 91 GB of 183 — because upstream weights are a 2-bit quant and "PolyStrata wasn't set up to run on cards this large" → "a really dumb config." Re-served Q8 (download 162 GB in 23 min), 256K context. Expert ratios quoted: GLM-53 Flash "320 vs 18", DS4 "550 vs 16" (garbled).

His fork's design rules (worth stealing as evaluation criteria)

  • Correctness oracle = match llama.cpp. Acceptance bar: within llama.cpp's own CPU-vs-GPU kernel disagreement floor. Step 12 achieved this for Flash Next IQ3XXS and Qwen 3.6 Q3KM only — he flagged that it "cheated" (no DeepSeek/GLM coverage yet).
  • No Python, no network at build: "delete all the Python, make it build offline" after rsync; compile llama.cpp's server files unmodified, replace context.cpp. Runtime-tunable knobs (temperature, top-K) via flags, not build time.
  • Engine agent recap: "a fork of PolyStrata 0.1.45… keeps upstream's two engine binaries (Strata and Strata-Poly)… C++ front-end built from the llama.cpp server at the engine's pin commit. No Python and no network at build or at start."
  • De-prioritised: Windows support, Dockerfile.
  • Main branch currently works only on a single GPU and only for Qwen 3.8 Flash Next; "unbiased strata two" targets ~5–6 model families. He pushed the branch at stream end (no URL given in transcript).

Model landscape mentioned (labels as heard — several garbled)

Qwen 3.8 Flash Next; "Qwen 3.6 35 PA 3B" (≈35B-A3B); "Qwen 3.6 Ornith/Ornate"; GLM "53 Flash"; DeepSeek "4/4.1 Flash"; LFM 2.5; Orpheus ("the same as Qwen 3.6"); "Gemma 4… 26 E2B"; "Teal Coder"/"OTL Coder"; MiniCPM; 27B Qwen dense; Coder variant (needs ~32 GB). Claim circulating online that he doesn't buy: "Qwen 3.8 27B equals Claude Opus 4.6." Quantisation-method aside: a lab he calls "DOS Labs, the Austrian bros" with "GSQRC" (likely garbled — probably GPTQ-family / GOT-QAT or similar) "massively outperforming Unsloth" — do not act on this without checking; names are garbled. PrismML "Bonsai 2" critique: re-quantised Qwen marketed as new models ("releasing Qwen fine-tune quants… claiming it as a new model release" — chat, quoted).

Harness / practical wisdom (chat reports labelled)

  • Compaction: Discriminatory David (chat): "most harnesses need 10–15% headroom for compaction; 90% triggers it." Hermi (chat): compacts in two chunks — Bob expects that to lose tool-call context. Rant: debugging compaction "with 90 gigs of VRAM available" and a 32K context = config error, not model error.
  • Bob has used Hermes but deliberately stays harness-light: "laser-focused on how good is this particular model at coding this particular gnarly thing without the noise of 20 different variants of a harness." (Earlier phase: "20 Opus threads reviewing each other's code" — abandoned.)
  • Harness-vs-backend bug pattern: harness times out and errors while the backend keeps processing — suspect the backend when it's freshly vibe-coded.
  • Renting GPUs: fine for learning; "voting with my wallet" but private benchmark tasks feel weird on rented boxes; RunPod literally has a secure-cloud vs community-cloud selector. "Customer data should never touch the public internet — one legit reason for running local" (chat, agreed with).

Personal/community bits

Announced challenge with Digital Spaceport ("Steve — No Place Like Local Host"): AI game-coding contest (Smash Bros, Stratego, chess, Brood War) with model-size restrictions (e.g. 4B cap). 720p no-ad/no-tracking WebTorrent downloads of all streams on his own site (YouTube rejected the link in a description as "not a valid video description"). RunPod billing scare: support deleted his pods mid-stream; ~$80/hr spend confusion vs his $16+$0.50 pods; spend limit $50. First time on stream: his abandoned local-AI-for-lawyers startup research (found lawyers happily send confidential data to cloud legal AI; calls zero-data-retention "the biggest misnomer"). Chat user "Run for Fun" claims a Strata fork with security fixes and that "Daybreak" found 66 vulnerabilities in Strata (unverified, chat claim).

Why this matters for a local Strata setup

  • Confirms a familiar pattern: Strata-family engines ≈120 tok/s vs llama.cpp 70–80 on comparable rented hardware, with a big prompt-processing advantage.
  • The two criticisms (markdown-as-install, runtime auto-update) are real supply-chain considerations for any local box: pin versions, keep the .verified.json SHA-256 manifests, and be deliberate about accepting auto-updates.
  • The Coder-variant result (~90% BBW but only equal to 35B-A3B) matches the "Coder needs 32 GB and isn't dramatically better" picture from earlier research.
  • His 10–15% compaction headroom figure lines up with Hermes' own compaction buffer behaviour.

All numbers above are as spoken in auto-captions; verify before quoting anywhere formal.