S-MoE

Appearance

The Multi-Protocol Server ​

serve.py wraps the engine in both industry protocols — OpenAI Chat Completions and Anthropic Messages — so any client that speaks either can talk to S-MoE: Open WebUI, LibreChat, Claude Code, editor plugins, the openai SDK, the anthropic SDK, plain curl. The engine gains an ecosystem without gaining a single dependency: the server is one file of Python stdlib (http.server) plus the transformers tokenizer the console already uses.

bash
.venv/bin/python serve.py --port 8000

Like the console, it is deliberately thin. The server owns HTTP, tokenization, and the chat template; the persistent smoe-engine --serve subprocess it spawns owns everything else. Between the two runs the same serve protocol the console speaks — the engine cannot tell them apart.

Endpoints ​

RoutePurpose
POST /v1/chat/completionsOpenAI — Chat completion, streaming (SSE) and non-streaming
POST /v1/messagesAnthropic — Messages API, streaming (typed-event SSE) and non-streaming
GET /v1/modelsEvery vault discovered in --vault-dir, the loaded one flagged active
GET /health{"status":"ok","engine":"up"} — liveness for supervisors
GET /sysmemmacOS physical-memory snapshot (installed / used / free, plus the engine's resident set) for the console's RAM meter
GET /A built-in browser chat console (see below)

OpenAI example ​

bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "s-moe",
    "messages": [{"role": "user", "content": "Explain in one sentence what a MoE model is."}],
    "temperature": 0.7,
    "max_tokens": 256,
    "stream": true
  }'

The model field accepts any id from /v1/models; unknown labels (including the legacy s-moe alias) simply run on whatever is loaded, so old clients keep working.

Or through the SDK:

python
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")

stream = client.chat.completions.create(
    model="s-moe",
    messages=[{"role": "user", "content": "Explain in one sentence what a MoE model is."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Anthropic example ​

bash
curl http://127.0.0.1:8000/v1/messages \
  -H 'Content-Type: application/json' \
  -H 'x-api-key: unused' \
  -H 'anthropic-version: 2023-06-01' \
  -d '{
    "model": "s-moe",
    "max_tokens": 256,
    "messages": [{"role": "user", "content": "Explain in one sentence what a MoE model is."}],
    "stream": true
  }'

Or through the SDK:

python
from anthropic import Anthropic
client = Anthropic(base_url="http://127.0.0.1:8000", api_key="unused")

with client.messages.stream(
    model="s-moe",
    max_tokens=256,
    messages=[{"role": "user", "content": "Explain in one sentence what a MoE model is."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

Using a Chat UI ​

For a real conversation you want a chat window, not curl. There are two paths, and the server supports both without any extra moving parts.

The built-in console. The server serves a single self-contained page at its own root — just open http://127.0.0.1:8000/ in a browser. It streams tokens as they arrive, shows the honest finish_reason (so a "length" cut-off is visible, not silent), and resends the full history each turn — which is exactly what keeps the engine's KV prefix-match warm. A model dropdown in the header lists every discovered vault (quantization beside each id, lineage in the tooltip); picking another one switches the engine on the next message, with an inline notice that the first reply pays the new vault's cold boot. Every reply carries a live turn monitor: a running wait counter that names the exact prompt size the moment it is known ("prefilling 128 tok prompt · 12.4s · cold start"), then time-to-first-token, token count, and decode tok/s updating as the answer streams — frozen on completion with the exact counts from the response's usage. The prompt count is the full templated history, the same figure the published benchmarks quote (a 45-token prompt), so a wait is immediately comparable to them. On a machine where an answer is measured in tens of seconds, seeing which phase the time went to is the difference between a hang and a prefill. A context-budget meter in the header tracks the same running figure against the engine's fixed KV window — 1,240 / 4,096 — filling and turning amber as the conversation grows, and posting a one-time notice at the wall, so the point where the window begins to slide (see below) is visible rather than silent. Beside it a system-RAM meter — fed by GET /sysmem — shows installed memory, current use, and the free headroom S-MoE's expert-streaming ring can grow into (14.0 / 48.0 GB · 34.0 free), coloured by macOS memory pressure rather than raw percentage so a box that legitimately runs high under a 235B stream isn't flagged as full; its tooltip breaks out wired, compressed, and the engine process's own resident set. It is one dependency-free HTML file (webchat.html, served same-origin so no CORS setup is involved), thin in the same spirit as the console and the server themselves. It exists to exercise the API by hand, not to be a product.

Any OpenAI or Anthropic client. Because the API is genuinely shaped for both protocols, third-party front-ends work unmodified:

  • Open WebUI — set an OpenAI connection with base URL http://127.0.0.1:8000/v1 and any non-empty API key; its model picker lists every discovered vault, and selecting one switches the engine.
  • LibreChat, editor plugins, the openai SDK — same base URL, same result.
  • Claude Code — the Anthropic coding CLI works natively: ANTHROPIC_BASE_URL=http://127.0.0.1:8000 ANTHROPIC_API_KEY=unused claude. No proxy, no translation layer — S-MoE speaks the Anthropic Messages API directly.

The built-in console proves the plumbing in ten seconds; a full client like Open WebUI proves the OpenAI compatibility; Claude Code proves the Anthropic compatibility. Neither adds a dependency to the engine.

The Model Fleet: Discovery and Switching ​

At startup the server scans --vault-dir (default vault/): every *.smoe with a sibling *.scout.safetensors is a servable model, its API id the filename stem. Identity comes from the vault's own bytes — quantization from the header, model_type from the arch block — so /v1/models reports what each vault actually is, not what a config claims:

json
{"id": "qwen3-30b-instruct", "active": true,  "quant": "Q4",
 "model_type": "qwen3_moe", "moe_layers": 48, "experts_per_layer": 128}

A request whose model names a non-active vault switches the engine: under the generation lock, the server shuts the current engine subprocess down and boots the named vault. Exactly one engine runs at a time — the machine cannot hold two — so the switching request pays that vault's cold boot before its prefill begins. Switching back is cheaper only in tokenizer terms (tokenizers stay cached); the engine boot is paid every time. Because switching rides the standard model field, it works from any client — OpenAI or Anthropic: Open WebUI's model picker switches vaults with no S-MoE-specific code, and so does setting the model field in a Claude Code session.

Each vault's tokenizer resolves to checkpoints/<stem>/ when that directory exists — the shatter names the vault after the checkpoint directory, so a freshly shattered model pairs automatically — falling back to --tokenizer otherwise. The tokenizer contract still applies per vault: the tokenizer must match the lineage the vault was shattered from.

One Context, Serialized Requests ​

The engine holds one conversation context and prefix-matches every request against it (the same LCP contract the console relies on). Two consequences follow, and the server is built around both:

  1. Generations are serialized. A single lock queues concurrent HTTP requests — whether they arrive via the OpenAI or Anthropic endpoint; each runs to completion before the next begins. There is no batching — the engine's throughput is bound by NVMe expert streaming, not by parallelism the hardware doesn't have. The lock also guarantees the engine pipe is at a request boundary before it is released: if a streaming client disconnects mid-response, the server drains the abandoned generation to its terminator rather than leaving its tokens in the pipe for the next request to misread. The engine still completes the abandoned generation (there is no cancel in the serve protocol), so the drain can hold the lock for as long as that generation runs — the engine would be busy for exactly that long either way.
  2. Continued conversations are fast. A client that resends the growing message list (which is exactly what OpenAI-style clients do) hits the KV-cache prefix match — only the new turn is prefilled. An unrelated conversation diverges early and pays a fuller re-prefill. Correct either way; only warm-up differs.

The KV window is a fixed 4096-token ring — prompt template, every user turn, and every reply share it, cumulatively. A conversation that grows past it does not error: the ring slides, dropping the oldest tokens, so the model gradually forgets the start of the exchange. The built-in console's context meter tracks this budget and warns once at the wall; resetting the conversation clears the window.

Where the template lives

The chat template is applied in the server, not the engine — the engine only ever sees token IDs. The tokenizer contract applies unchanged, per vault: each model's resolved tokenizer must match the lineage its vault was shattered from.

Per-Request Sampling ​

The standard sampling fields are forwarded to the engine per request, mapped onto the serve protocol's override fields — identically for both OpenAI and Anthropic requests:

Request fieldProtocol fieldEngine default
temperaturet=launch --temperature
top_pp=launch --top-p
top_k (extension)k=launch --top-k
repetition_penalty (OpenAI extension, also rep_penalty)r=launch --rep-penalty

Only fields a request actually sets are sent; the engine's launch flags cover the rest, and an override never outlives its request. temperature: 0 selects the engine's pure greedy path — deterministic, bit-identical output for the same conversation.

max_tokens (or max_completion_tokens) is honoured per request and capped at 4096.

finish_reason / stop_reason, Honestly ​

The engine reports why generation ended — a fin= trailer ahead of <<DONE>> on the wire — and the server passes it through in protocol-native terms:

OpenAI finish_reasonAnthropic stop_reasonMeaning
"stop""end_turn"The model sampled an EOS token — the reply is complete.
"length""max_tokens"The max_tokens budget ran out mid-thought.

Clients that auto-continue on length (many chat UIs do) therefore work correctly instead of silently truncating.

Token accounting is exact in both modes: non-streaming responses carry the standard usage object (OpenAI: prompt_tokens/completion_tokens, Anthropic: input_tokens/output_tokens), and streaming responses attach the same counts to their final events.

Launch Flags ​

bash
.venv/bin/python serve.py \
  --host 127.0.0.1 --port 8000 \
  --vault-dir vault \
  --vault vault/<stem>.smoe \
  --tokenizer Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --ring 0 --workers 4 \
  --temperature 0.6 --top-p 0.95 --top-k 50 --rep-penalty 1.1 \
  --max-tokens 256
  • --vault-dir — the fleet directory: every *.smoe + *.scout.safetensors pair here is servable.
  • --vault — the vault loaded at startup (its scout is derived from the stem; --scout overrides).
  • --tokenizer — the fallback for vaults without a matching checkpoints/<stem>/ directory.
  • --temperature / --top-p / --top-k / --rep-penalty — the engine's session defaults, used for any sampling field a request omits.
  • --max-tokens — the default completion budget when a request doesn't set one.
  • --ring 0 — delegate ring sizing to the engine's auto-tuner, as with the console.
  • --model-id — a legacy alias: requests naming it run on the active model.

The engine boots in the background after the tokenizer loads; the first request simply waits for it. If the engine dies — OOM kill, Metal failure — the next request respawns it and re-prefills from scratch: slower, but correct, and the server never goes down with its engine.

Setting Expectations ​

The server changes reach, not speed. Decode remains NVMe-bound (see Optimizations) — roughly a word per second on the 235B with 48 GB hardware, with a prefill pause before the first token; the Qwen3-30B vault achieves up to 14 tok/s and is better suited for interactive or coding workflows. Streaming ("stream": true) is the recommended mode: tokens appear as they are generated, so the wait reads as deliberation instead of silence.

Current limitations: one vault loaded at a time (switching restarts the engine), no authentication, binds to 127.0.0.1 by default. Put a reverse proxy in front if it must leave localhost.

MIT License.

Copyright © 2026 S-MoE / Luca Visciola