Direct I/O & Streamer

The Streamer is S-MoE's relentless, invisible lifeblood. It is the pool of I/O worker threads that pulses permanently in the background, moving expert blobs from the .smoe vault into the lock-free ring buffer. It serves two masters in strict order: demand — the exact experts the current layer is spinning on — and, only when demand is silent, speculation — the deeper gate ranks bet on the next token.

---

The Direct I/O Principle

Standard file I/O on macOS forces all reads through the OS kernel's virtual memory page cache. For chaotic workloads, this is fine. For S-MoE, it is a catastrophic weakness:

1. It forces a sluggish copy from kernel space to user space (double-buffering).
2. It competes desperately with the OS for Unified Memory.
3. It inflicts jagged, unpredictable latency spikes during GC sweeps.

S-MoE obliterates this bottleneck using pure F_NOCACHE:

// Inscribed on every vault file descriptor at open time
::fcntl(fd, F_NOCACHE, 1);

With F_NOCACHE engaged, pread() transforms into a majestic, unfiltered direct DMA transfer:

NVMe SSD → DMA controller → Unified Memory (pre-allocated, 16KB-aligned)

No kernel copy. No page cache taxation. The data materializes exactly where the Metal kernel will effortlessly consume it.

16 KB Alignment is Absolute

F_NOCACHE demands that all destination buffers perfectly align to the Apple Silicon hardware page boundary. S-MoE enforces this ruthlessly with posix_memalign(16384) for every ring buffer slot at startup. Misalignment guarantees a bus error.

---

Ring Buffer Architecture

The Streamer orchestrates a pre-allocated ring of N slots. Each slot is sized to harbor the vault's largest expert blob, rounded up to the 16 KB page (for Qwen3-235B at Q4: ~10 MB per slot).

Sizing is automatic by default (--ring 0 or omitted): at startup the engine scans the expert table for the true maximum padded_size, then budgets subtractively — the OS's pre-pressure availability estimate (kern.memorystatus_level) minus the Scout file, a fixed engine overhead, and an OS floor of max(RAM/8, 4 GB) — clamped to [64, 4096] slots and floored at a conservative 25%-of-free fallback. On a 48 GB machine this yields ~15 GB of ring. Override the slot count with --ring N or the slot size with --slot-mb N — but the auto-tuner is the only setting that survives a change of quantisation depth, since Q2 blobs are half the size of Q4 blobs.

Ring Buffer (pre-allocated at startup, existing purely in UMA)
┌──────────┬──────────┬──────────┬──────────┬──────────┐
│  Slot 0  │  Slot 1  │  Slot 2  │  ...     │  Slot N  │
│ READY    │ LOADING  │ FREE     │ ...      │ CONSUMED │
└──────────┴──────────┴──────────┴──────────┴──────────┘
     ▲            ▲                               │
     │            │                               │
  GPU reads   Streamer                         Released
  from here   pread()                          to FREE

The Fluid State Machine — a True LRU Cache

Each slot flows through four states using std::atomic compare-exchange. We banish mutexes. We reject condition variables.

EMPTY ──► LOADING ──► READY ──► CONSUMED ──┐
              ▲                            │ retained (data stays valid)
              └──── LRU eviction ◄─────────┘

The crucial property: CONSUMED is retention, not disposal. When the GPU releases a slot, its expert data remains intact and its (layer, expert) identity remains registered. A later request for the same expert re-claims the retained slot as a cache hit — zero bytes read. Expert weights are immutable, so a retained slot can never go stale — retention survives across tokens and across serve-mode turns.

Capacity is reclaimed lazily by prefetch() itself: when no EMPTY slot exists, it evicts the least-recently-used idle slot — and only an idle one. Three rules keep this safe and effective:

1. A slot with ref_count > 0 is untouchable — the GPU may still be reading it; evicting it would let pread() overwrite data mid-kernel.
2. Freshly loaded slots are stamped with a fresh LRU tick — otherwise a large prefetch burst inherits stale ticks and evicts its own head before it is ever read.
3. There is no wholesale pruning. Anywhere. Lazy LRU is the single eviction discipline in both regimes. Any per-token wipe would destroy the measured 46.4% adjacent-token expert overlap that retention converts into skipped reads.

Two Queues, One Law: Demand First

prefetch(layer, expert, speculative) routes requests through one of two identical lock-free MPMC queues (Vyukov's algorithm, power-of-two capacity, zero allocation). Workers pop the demand queue first, always; the speculative queue is touched only when demand is empty. A burst of next-token bets can therefore never starve the fetches the current layer is spinning on.

Speculation, honestly measured

At Q4 on the 48 GB reference machine, speculative prefetch of gate ranks 9–16 (--spec 8) lifts ring coverage from 46.4% to 63.7% — and still makes the engine slower (+242 ms/token of dense-path slowdown from memory-bandwidth contention, +144 ms/token of io-spin from worker occupancy: priority ordering cannot preempt an in-flight 10 MB pread). The default is --spec 0. The machinery is correct, tested, and waiting for Q2 vaults, where every speculative byte costs half as much and the bandwidth it feeds on is twice as idle.

The Worker Threads

The Streamer commands --workers background I/O threads (engine default: 16; the chat console dials it to 4). Each sovereign worker, on its own private F_NOCACHE file descriptor (macOS serialises Direct I/O per descriptor — one fd each unlocks full NVMe queue depth):

1. Pops the demand queue; only if it is empty, the speculative queue.
2. Receives a slot already claimed LOADING by prefetch().
3. Executes pread(fd, slot.data, blob_size, expert_offset) — an unyielding transfer.
4. Elevates the slot to READY with a release store.

The main thread elegantly invokes streamer.claim_specific(layer_id, expert_id)—a non-blocking poll that returns either a freshly READY slot or a retained cache hit (an idle CONSUMED slot still holding this expert's data). If it returns nullptr, it spins briefly, knowing the Streamer is already racing to deliver the payload.

---

Expert Layout Cache

At dawn, the Streamer ingests the entire expert table from the vault header, weaving an instantaneous in-memory lookup map:

std::unordered_map<uint64_t, ExpertEntry> expert_index;
// key = (layer_id << 32) | expert_id

This guarantees blistering O(1) lookup of any expert's byte_offset and padded_size. No per-token scanning. Just pure knowledge.

---

Popularity-Ordered Pre-Warming

The engine learns which experts the world actually asks for. Every successful claim increments a per-(layer, expert) histogram, persisted to vault/expert_freq.bin across sessions. A background thread streams the historically hottest experts into the ring whenever the engine is idle:

• At startup, before the first token drops.
• Between serve-mode requests — while the human types, the ring quietly re-warms with the conversation's own favourite experts.
• Never during a request: an atomic engine_busy flag pauses the trickle instantly, so demand reads always own the NVMe queue. A whisper-quiet 4 ms sleep between prefetches keeps the queue shallow even when active.

The prewarm budget reserves headroom for the live working set — max(128, ring/4) slots — and the thread is joinable, never detached: it is cleanly stopped and joined at shutdown so it can never outlive the ring it feeds.

An honest measurement

Even at the subtractive budget's ~15 GB ring (48 GB machine, Q4 vault), a single turn's expert working set (~70 GB of per-layer unions) dwarfs the cache, so prewarm's measured effect on turn-1 prefill latency is neutral. It tops up a cache that decode hits at ~46% between turns; its full moment arrives with Q2 vaults, which double the slot count. The histogram is already recording.