System Overview
S-MoE is a model-agnostic inference engine for fine-grained Mixture of Experts language models. It is built on one radical premise: memory is a transient execution medium, not a storage pool.
The Monolithic Delusion
Every conventional local LLM runtime — llama.cpp, Ollama, MLX — operates on the same flawed assumption: the entire model must reside in RAM. For a 235B parameter model, this requires ~470 GB of Unified Memory. No consumer Mac ships with more than 192 GB.
S-MoE rejects this assumption completely.
A Mixture of Experts model activates only a tiny fraction of its parameters per token — typically 8 experts out of 128, across 94 MoE layers. The remaining 95%+ of the model is dead weight, sitting unused. S-MoE exploits this sparsity radically: keep the entire expert vault cold on the NVMe SSD, and stream only the experts that will actually fire — before they are needed.
The model never fits in memory. It doesn't need to.
The Four-Component Architecture
┌─────────────────────────────────────────────────────────────────┐
│ SCULPTOR (offline) │
│ shatter_moe.py │
│ Any MoE .safetensors → .smoe vault + .scout.safetensors │
│ Auto-detects: d_model, vocab_size, ffn_dim, MoE topology │
└────────────────────────┬────────────────────────────────────────┘
│ One-time offline transform
│
┌───────────────┼────────────────────────┐
▼ ▼ ▼
┌────────────────┐ ┌──────────────┐ ┌────────────────────────────┐
│ SURFACE SCOUT │ │ STREAMER │ │ METAL KERNEL │
│ scout.cpp │ │ streamer.cpp │ │ kernels.metal │
│ │ │ │ │ │
│ Dense backbone │ │ F_NOCACHE │ │ SMOE-Q2/Q4 fused │
│ resident in │ │ pread() into │ │ dequant + FFN on │
│ UMA: embeds, │ │ lock-free │ │ Metal MTLBuffers │
│ attention, │ │ LRU ring │ │ (zero-copy UMA) │
│ norms, router │ │ cache (two │ │ │
│ gates, LM head │ │ prio queues) │ │ │
└───────┬────────┘ └──────┬───────┘ └────────────┬───────────────┘
│ weights │ aligned blobs │ output vectors
└──────────────────┴───────────────────────┘
│
main.cpp conductor
request loop · LM head · sampling
(boot machinery: engine.cpp · protocol: serve.hpp)
│
┌────────────┴────────────┐
prefill.cpp decode.cpp
layer-major token-serial step,
batched prefill exact-routing decodeTwo Regimes, One Engine
The engine treats the prompt and the reply as fundamentally different problems:
- Prefill (the prompt — the future is known): tokens move through the model layer by layer in chunks, routing is computed exactly from the true hidden state, and each layer's deduplicated expert union is read from the vault once per chunk — I/O stops scaling with prompt length. In server mode, only the suffix beyond the previous conversation is prefilled at all. See Layer-Major Batched Prefill.
- Decode (generation): routing is computed exactly here too — the real router gate evaluated on the heavy hidden state at every MoE layer. The future is covered not by a separate predictor but by the routing archive: the ring retains every recently-read expert as an LRU cache, and adjacent tokens reuse 46% of each other's experts (measured). Prediction earned its retirement — see below.
What Happens at Each Generated Token Step
At every generated token step, one heavy pass walks the layers while the Streamer works ahead of it:
| Actor | Action | Hardware |
|---|---|---|
| Heavy pass | Per layer: the full attention block (QKV → QK-norm/RoPE/KV-append → GQA attention → o_proj) in one Metal command buffer, one sync, then a NEON 128 × d_model gate matvec on the true hidden state → the layer's exact top-8 experts and weights (norm_topk_prob per the Qwen3 spec) | Metal + CPU |
| Streamer | Demand-fetches this layer's misses; retained ring slots cover ~46% of each layer's experts from the previous tokens' reads. Optional --spec N fires gate ranks 9…8+N through a low-priority queue as next-token bets | NVMe SSD → UMA (worker threads) |
| Metal Kernel | Executes the SMOE-Q4 fused FFN for each claimed expert — coalesced simdgroup-per-row kernels running at DRAM bandwidth — weighted-summed into the residual | Metal GPU |
Why there is no routing predictor
A predictive pass would have to beat the free oracle, and none does: the retained ring covers 46.4% of every token's experts at zero cost, the previous token's own gate ranking covers 63.7% — while a full dense pass evaluated as a predictor measures 51.5% at ~265 ms/token, its mispredicted prefetches queuing ahead of demand reads. Decode therefore runs the true Qwen3 routing, exactly: 1.43 t/s measured on the 48 GB reference machine (Optimizations).
Memory Budget
The budget scales with the target. For Qwen3-235B-A22B-Instruct-2507 (the verified frontier configuration):
| Component | Footprint |
|---|---|
| Surface Scout weights (dense backbone, bfloat16) | ~16 GB |
| Expert ring buffer (auto-sized: available RAM minus Scout + OS reserve, 64–4096 slots × ~10 MB Q4 slot) | ~2–20 GB, adapts to the machine (~15 GB on 48 GB) |
| KV-cache (4K context, GQA 4 KV heads × 128 dims, 95 layers) | ~1.6 GB |
| Metal working memory | ~512 MB |
| OS + applications | ~4 GB |
| Expert vault (~117 GB Q4) | Cold on NVMe — 0 bytes of RAM |
The expert vault — which is 95%+ of the model by parameter count — occupies exactly zero bytes of RAM at any time. It is streamed on demand from the SSD.
Ring sizing is fully automatic: at startup the engine scans the expert table for the true maximum blob size, then budgets subtractively — the OS's own available-memory estimate minus the Scout, engine overhead, and an OS floor (clamped to [64, 4096] slots, overridable with --ring N). This is why the practical floor for the 235B frontier is 32 GB of Unified Memory — the Scout alone claims half of it — while smaller fine-grained MoE vaults run comfortably in 16 GB.