Quick Start

The era of renting intelligence is over. You do not need to lease an H100 in the cloud for $15,000 a month to glimpse the frontier. This is about emancipating frontier intelligence and placing it directly onto the elegant MacBook on your desk.

Already bootstrapped with Setup? The full path from model download to a live chat session is below. S-MoE embraces any supported MoE model—consult the Supported Models for your arsenal.

---

The Path to Liberation

1. Download the monolith

Authenticate with the old world, then pull the checkpoint. The frontier target is Qwen3-235B-A22B-Instruct-2507 — the non-thinking instruct variant. This matters: the vault and the chat template must come from the same lineage (see the warning below).

.venv/bin/hf auth login

# Frontier: Qwen3-235B-A22B-Instruct-2507 — a ~470 GB descent into your drive.
# An external SSD works fine; only the shattered artifacts need to live on fast NVMe.
.venv/bin/hf download Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --local-dir ./checkpoints/qwen3-235b-instruct \
  --include "*.safetensors" "config.json" "tokenizer*"

Name the directory deliberately

The Sculptor names its artifacts after the checkpoint directory: shattering ./checkpoints/qwen3-235b-instruct produces qwen3-235b-instruct.smoe and qwen3-235b-instruct.scout.safetensors. Choose the directory name you want your vault to carry.

2. Probe — let the engine map the topology

make probe MODEL=./checkpoints/qwen3-235b-instruct

A dry run. No files are written. The Sculptor auto-detects d_model, vocabulary, FFN dimensions, MoE layer count, and experts per layer, then estimates the output sizes.

3. Shatter — quantise the monolith into the vault

make shatter MODEL=./checkpoints/qwen3-235b-instruct OUT=./vault

Integrity validation (--validate) is always included by the Make target. Quantisation defaults to SMOE-Q4 (4-bit); pass ARGS="--bits 2" for the smaller, lossier SMOE-Q2. Expect the full 235B shatter to take hours and the Q4 vault to weigh ~117 GB.

4. Forge the engine

make all

5. Ignite

The recommended interface is the chat console, which handles the chat template, token streaming, and BPE-delta decoding:

.venv/bin/python chat.py

chat.py spawns build/smoe-engine once, in persistent server mode (--serve), and keeps it alive for the whole session — the KV-cache survives across turns, so every message after the first prefills only its new tokens (see The Chat Console). The --vault and --scout paths are hardcoded in its engine command — align them with your chosen checkpoint directory name, and make sure its AutoTokenizer.from_pretrained(...) points at the same model you shattered.

For raw, single-shot invocations (scripting, benchmarks), command the engine directly:

./build/smoe-engine \
  --vault vault/qwen3-235b-instruct.smoe \
  --scout vault/qwen3-235b-instruct.scout.safetensors \
  --tokens-in "151644,872,198,9707,151645,151644,77091,198" \
  --tokens 200 \
  --raw-ids

--tokens-in takes pre-tokenized IDs (produced by the matching tokenizer); --raw-ids streams raw token IDs on stdout for a wrapper to decode. Ring sizing is automatic — the engine measures the vault's real blob size, asks the OS how much memory is available before pressure, reserves the Scout and an OS floor, and gives the rest to the ring (override with --ring N).

The template must match the vault

Qwen3 ships in two lineages: the original thinking model (Qwen3-235B-A22B) and the non-thinking Instruct-2507. The thinking template injects <think> tokens whose embeddings are untrained in the Instruct-2507 weights — feeding them to an Instruct-2507 vault collapses output into degenerate punctuation. Always load the tokenizer from the same checkpoint you shattered.

---

The Artifacts of make shatter

When the monolith is successfully shattered, two perfectly optimized artifacts manifest in your output directory:

File	Description
<stem>.smoe	The Vault — all routed expert blocks, ruthlessly SMOE-Q4 quantised, aligned to the 16 KB page boundary.
<stem>.scout.safetensors	The Surface Scout — the dense backbone: embeddings, attention, norms, and routing gates, saved as-is in bfloat16.

The .smoe vault rests cold on the NVMe SSD, vast and silent. The .scout.safetensors ascends entirely into Unified Memory at startup, an ever-present sentinel guiding the session. For Qwen3-235B the Scout weighs ~16 GB — plan your RAM accordingly (see Setup).

---

Testing on a Fragment

If you wish to test the waters before shattering the entire model, validate the pipeline on a subset:

make shatter \
  MODEL=./checkpoints/qwen3-235b-instruct \
  OUT=./vault-test \
  ARGS="--max-layers 2 --max-experts 8 --measure-error"

This surgical strike shatters only the first 2 MoE layers × 8 experts each, printing exact reconstruction error metrics. It is the perfect sanity check before committing to the full awakening.