The Chat Console

chat.py is the human-facing gateway to the engine — a minimal, dependency-light REPL that turns the raw token stream of smoe-engine into a conversation. It is deliberately thin: the Python layer owns tokenization and display, the C++ engine owns everything else.

bash

.venv/bin/python chat.py

One Engine, Many Turns

The console spawns smoe-engine once, in persistent server mode (--serve), and keeps it alive for the entire session. The engine's KV-cache, expert ring cache, and Scout state survive from turn to turn — so a follow-up message pays only for its new tokens, never for the conversation that came before.

Turn 1:  [full prompt prefill]           ──►  first token in ~30 s (cold)
Turn 2+: [suffix-only prefill]           ──►  first token in ~14 s
         (the entire prior conversation is already in the KV-cache)

Without a persistent process, every turn would fork a fresh engine, re-page 16 GB of Scout weights, and re-prefill the whole history from token zero — a cost that grows with every exchange. Server mode is what makes turn latency independent of conversation length.

Division of Labour

Concern	Owner	Why
Chat template, tokenization	`chat.py` (HuggingFace `AutoTokenizer`)	Templates are model-lineage-specific and evolve quickly; Python is the right altitude.
Multi-turn history	Both, by contract	The console sends the full templated token stream every turn; the engine prefix-matches it against its own stored stream and prefills only the unseen suffix.
Inference, sampling, streaming, KV persistence	`build/smoe-engine`	The sovereign C++ core. One process per session.
BPE-delta decoding	`chat.py`	Token IDs stream back via `--raw-ids`; the console decodes incrementally and only prints once a valid UTF-8 boundary is reached (no `�` artifacts on multi-byte characters).

The Serve Protocol

The console speaks to the engine over plain stdin/stdout lines:

──► GEN <max_tokens> [ovr ...] <id0>,<id1>,...  one request per turn
◄── 32 386 12735 ...                            raw token IDs, streamed
◄── fin=<reason> <<DONE>>                       end of reply

──► RESET                                       drop all cached state
◄── <<DONE>>

◄── <<ERR bad_request>> │ <<ERR ctx_overflow>> │ <<ERR empty_suffix>> │ <<ERR suffix_too_long>>

fin=<reason> reports why generation ended — stop (the model sampled an EOS token) or length (the max_tokens budget ran out). It rides ahead of <<DONE>> as a key=value token, mirroring the request-side override syntax, so clients that only understand numbers and sentinels (the console among them) skip it without ceremony.

[ovr ...] are optional per-request sampling overrides — space-separated key=value fields between <max_tokens> and the ID list:

field	meaning	default
`t=<f>`	temperature (`0` → greedy)	launch `--temperature`
`p=<f>`	top-p	launch `--top-p`
`k=<u>`	top-k (capped at 1024)	launch `--top-k`
`r=<f>`	repetition penalty	launch `--rep-penalty`

The first field without = starts the ID list, so clients that send the plain form are parsed unchanged; unknown keys are ignored. Launch flags remain the session defaults — an override applies to one request only.

The ID list is always the entire conversation as produced by apply_chat_template. The engine computes the longest common prefix against its stored stream:

Stream extended (the normal case): only the suffix is prefilled. The engine's stream includes its own previously generated tokens, so the match runs straight through the assistant's last reply.
History diverged (edited or regenerated): the cached KV beyond the divergence is invalid — the engine resets and prefills from scratch. Correctness never depends on the cache.

This prefix contract makes the protocol robust to chat-template quirks by construction — the console never needs to compute deltas.

Engine launch flags

bash

build/smoe-engine \
  --vault vault/<stem>.smoe \
  --scout vault/<stem>.scout.safetensors \
  --serve \
  --ring 0 \
  --workers 4 \
  --temperature 0.6 --top-p 0.95 --top-k 50 --rep-penalty 1.1 \
  --raw-ids

--serve — persistent server mode. Without it, the engine runs a single --tokens-in request and exits (still supported for scripting and benchmarks).
--ring 0 — delegates ring sizing to the engine's auto-tuner. The engine scans the vault for the true maximum expert blob size (Q2 blobs ≈ half the size of Q4 blobs) and budgets subtractively: the OS's own available-memory estimate (kern.memorystatus_level), minus the Scout file, a fixed engine overhead, and an OS floor — clamped to [64, 4096] slots, never below a conservative 25%-of-free fallback. A fixed slot count tuned for one quantisation level will over- or under-allocate on another; auto is the only durable setting.
--raw-ids — the engine emits bare token IDs on stdout. All human-readable text is produced by the console's incremental decode. Engine diagnostics live exclusively on stderr.

Two further flags exist for measurement and experimentation (not used by the console): --instrument prints a per-request decode timing breakdown on stderr — the buckets every optimization decision is priced by — and --spec N enables speculative prefetch of the next token's likely experts (default 0: measured net-negative at Q4, held in reserve for Q2 vaults).

Per-request max_tokens rides in the GEN line itself.

Failure Surfacing

The engine is mortal; the session is not. If the engine dies mid-turn — OOM kill, Metal failure, a bug — the console:

Prints [engine died (exit N) — restarting] and respawns it.
Retries the request once. The fresh engine has an empty stream, so the prefix match falls through to a full re-prefill — slower, but correct.
If the turn still yields no tokens, drops the failed user turn from the conversation history, so a crashed round cannot poison the context of subsequent turns.

Ctrl-C during generation kills and respawns the engine (there is no per-request cancel in the protocol yet) — the next turn re-prefills from the template.

The Tokenizer Contract

One lineage, end to end

The tokenizer loaded by chat.py must come from the same checkpoint that was shattered into the vault. Qwen3's two lineages are mutually treacherous:

Lineage	Template behaviour
`Qwen3-235B-A22B` (thinking)	Injects `<think>…</think>` tokens (IDs 151667/151668) via `enable_thinking`
`Qwen3-235B-A22B-Instruct-2507` (non-thinking)	No think block, no `enable_thinking` switch

The <think> embeddings are untrained near-zero rows in the Instruct-2507 weights (L2 ≈ 0.008 vs ≈ 1.2 for ordinary tokens). Injecting them mid-prompt collapses the residual stream: the model answers with degenerate punctuation and an early EOS. If your output looks like : or !!, audit the template before questioning the heavy engine.

Sampling Defaults

The console follows the Qwen3 model card recommendations for non-thinking mode:

Parameter	Value
`--temperature`	0.6
`--top-p`	0.95
`--top-k`	50
`--rep-penalty`	1.1 (last 256 tokens; common punctuation, `\n`, and EOS IDs are exempt)

Setting --temperature below 1e-4 switches the sampler to pure greedy argmax — useful for deterministic debugging (the engine seeds its RNG at 1337).

The Chat Console ​

One Engine, Many Turns ​

Division of Labour ​

The Serve Protocol ​

Engine launch flags ​

Failure Surfacing ​

The Tokenizer Contract ​

Sampling Defaults ​