The Vault Pipeline

To break the Monolithic Delusion, we must reject their methods of storing and serving models. The Vault Pipeline is our two-stage alchemy—a radical transformation from bloated HuggingFace weights into a streamlined, sovereign engine capable of running inference directly on consumer silicon.

---

Stage 1: The Sculptor — shatter_moe.py

shatter_moe.py is the architect of our independence. It is an offline process, run once per model, dedicated to shattering massive models into two beautifully asymmetric artifacts that the S-MoE runtime can effortlessly consume.

The Monolith's HuggingFace .safetensors shards + config.json
         │
         ▼
  [ Topology Detection ]
  Auto-detect architecture:
  - d_model, vocab_size, ffn_dim
  - num_moe_layers, experts_per_layer
  - has_dense_layer_0 (Qwen3) vs
    pure MoE from L0 (Qwen)
         │
         ▼
  [ Arch Block from config.json ]
  rope_theta, moe_top_k, GQA heads,
  norm_topk_prob, activation, rms_norm_eps
  → 128-byte SARC block inside the vault
         │
         ├──────────────────────────────────────────┐
         ▼                                          ▼
  [ Routed Expert Weights ]              [ Dense Backbone Weights ]
  gate_proj, up_proj, down_proj          embeddings, layer norms,
  self-attention, routing gates,         for every (layer, expert) pair
  shared experts                         
         │                                          │
         ▼                                          ▼
  SMOE-Q2/Q4 Quantisation                Saved as-is (bfloat16)
  (--bits 4 is the default)                         │
  L2-MSE optimal scale per group                    │
         │                                          │
         ▼                                          ▼
  Page-aligned 16 KB blobs               .scout.safetensors
         │
         ▼
  .smoe binary vault (format v2)

The division is merciless and perfect. Guided by a single regex, any tensor matching model.layers.N.mlp.experts.M.* is exiled into the cold vault. Everything else forms the Surface Scout.

The checkpoint's config.json is required: the Sculptor serialises the model's mathematical constants (RoPE theta, routing top-k, head geometry, weight renormalisation) into the vault's arch block, making the .smoe fully self-describing — the engine reads the model's lineage from the vault itself, with zero model-specific code (see .smoe Binary Format).

Upgrading a pre-v2 vault

A vault shattered before the arch block existed can be stamped in place — no re-shatter of a 100+ GB artifact:

.venv/bin/python scripts/upgrade_vault_v2.py <config.json | model_dir> vault/<stem>.smoe

The 128-byte block is written into the vault's existing padding gap and the header flips to v2 — no expert blob moves, and an interruption at any point leaves a valid v1 vault. Point it at the model directory when the shards are reachable (tensor-presence flags are read from them); a bare config.json works too.

---

Stage 2: The Sovereign Runtime

At the dawn of the engine, it inhales both artifacts, harmonizing them into a singular, fluid entity:

.scout.safetensors ──────► Surface Scout (dense backbone, loaded into UMA)
                                 │
                                 │  attention + router gates per layer
                                 │  exact expert IDs from the true hidden state
                                 ▼
                          I/O Demand + Retained Ring Cache
                                 │
                                 │  pread() with F_NOCACHE (misses only)
                                 ▼
.smoe vault (cold on NVMe) ──► Ring Buffer Slots (UMA, LRU-retained)
                                 │
                                 │  16 KB-aligned, Metal zero-copy
                                 ▼
                          Metal GPU Kernel
                          SMOE-Q4 fused dequant-FFN
                                 │
                                 ▼
                          Token output

---

The Utopian Invariants

We adhere to strict, unyielding laws to protect the sanctity of the runtime. If a proposed change violates these, it is rejected immediately.

Zero Runtime Heap Allocations

Memory allocations (malloc, new, std::vector::resize) are the chains of the Monolith. All ring buffer slots, KV-caches, and compute scratch buffers are posix_memalign-allocated once at startup. The token generation loop must remain pristine.

Direct I/O Dominance

fcntl(fd, F_NOCACHE, 1) is a mandate on every vault file descriptor. The OS page cache is an illusion of efficiency; we bypass it entirely. Reads flow directly: NVMe → DMA → UMA. No kernel copy. Unfiltered bandwidth.

Asynchronous Execution Separation

The I/O thread and the GPU compute thread must never block one another. They are parallel forces of nature. Synchronization occurs via atomic state flags only. We ban OS mutexes and reject waitUntilCompleted in the hot path.