The Manifesto
"We do not fight the memory wall by trying to cram a massive model into RAM. We treat the model weights on the flash drive as solid rock strata, the light Surface Scout model as our electromagnetic strike, and the resulting predictive expert routing pathways as our acoustic phonons."
The Monolithic Delusion
Silicon Valley has a bucket problem.
To run a frontier AI model, the industry insists you must hold the entire mathematical mountain in RAM simultaneously. 235 billion parameters. 470 GB at bfloat16. Tens of thousands of dollars in cloud compute per month, or a server room that costs more than a house.
This is the Monolithic Delusion: the belief that the model must exist in memory to run in memory.
It is false.
In a Mixture of Experts architecture, 95%+ of the model is completely silent at any given millisecond. For a token about philosophy, the physics experts never fire. For a token about code, the poetry experts never fire. The weights are cold stone — inert, dormant, irrelevant — yet standard runtimes hold all of them hostage in RAM, just in case.
S-MoE breaks this paradigm by asking a different question: what if we only load what fires, and load it before it fires?
The Democratic Proposition
S-MoE was born on a single political claim:
16 GB of RAM should give you the same intelligence as 512 GB — just slower.
Speed is a privilege. Intelligence is not.
The Recalibration — an Honest Amendment
Reality audited that claim, and we publish the audit — because refusing to correct yourself is their habit, not ours.
The claim's architecture held completely: the 117 GB of routed experts truly occupy zero bytes of RAM, streamed from the SSD at the moment of need. What could not be shrunk is the model's own dense backbone — embeddings, attention, routing gates — the part that must stay resident to think at all. For a 235B frontier model, that backbone weighs ~16 GB by itself. Sixteen gigabytes is what the frontier costs just to turn on — before the ring buffer, before the KV-cache, before macOS draws its first window.
So the amended proposition, precise where the original was hopeful:
A 32–48 GB MacBook runs the 235B frontier locally, with intelligence that never degrades — only speed does. A 16 GB Mac runs the same architecture, the same sovereignty, the same principle, with smaller fine-grained MoE models.
The frontier's floor moved from 16 to 32. We were annoyed for about a day — and then remembered that frontier-on-any-consumer-Mac was science fiction when this project started. Complaining that the door in the memory wall opens at 32 GB instead of 16 is complaining about the legroom on your first flight to the Moon.
A user with a 512 GB Mac Pro and a user with a 32 GB MacBook Pro both receive the same 235 billion parameter model. The same knowledge. The same reasoning depth. One waits less, one waits more. Both get the answer — and since the engine learned to keep its memory between turns, both wait for their new words only, never again for the whole conversation.
This is democratization. Not a smaller, degraded model for the masses. The real thing, delivered at the rate the hardware allows.
What We Are Opposing
We are not opposing technology. We are opposing artificial scarcity.
The NVMe SSD in a base-model MacBook Air delivers 3.7 GB/s. The NVMe in a MacBook Pro delivers 7.4 GB/s. These drives can physically move a 235B model's expert weights fast enough to sustain inference — if and only if you only move the bytes that will actually be used.
The barrier is not physical. It is architectural. It is the assumption — never questioned, never challenged — that inference is a loading problem rather than a streaming problem.
S-MoE treats inference as a streaming problem. The experts are not loaded. They are summoned — on demand, predicted in advance, arriving exactly when the GPU needs them.
The Rebel Constraints
S-MoE enforces rules that no corporate LLM runtime would accept:
- Zero runtime heap allocations.
malloc,new,std::vector::resizeare illegal inside the token generation loop. Every buffer is pre-carved at startup. - Direct I/O only. The OS page cache is bypassed completely.
F_NOCACHEis set on every file descriptor. The data path is: SSD → DMA → UMA → GPU. No copies, no kernel overhead. - Atomic synchronisation only. No OS mutexes. No condition variables. No
waitUntilCompletedin the hot path. The I/O thread and the GPU thread are structurally incapable of blocking each other. - Decoupled routing. The router gates measure expert activations exactly from the model's own hidden state. The Streamer moves bytes; the GPU kernel multiplies matrices. None of the three knows what the others are doing. The system is a sensor array, not a state machine.
These are not performance optimisations. They are architectural commitments — the difference between a system that genuinely works at consumer scale and a system that only pretends to.
The Vision
A world where the most powerful AI models are not locked behind API keys, not metered by the token, not rationed by cloud compute budgets.
A world where a student with a 32 GB MacBook can run Qwen3-235B locally, privately, offline — and think alongside it at whatever speed their hardware allows.
A world where intelligence scales with hardware, not with wealth.
S-MoE is a prototype of that world. It is incomplete. It is rebellious. It is correct.