Model Agnosticism

S-MoE was originally forged to run a single model—DeepSeek-MoE-16B. But the vision of liberation requires ultimate flexibility. We executed a radical architectural refactor to transform it into a model-agnostic inference engine. It now automatically, intuitively adapts to any fine-grained MoE architecture the moment it boots—today's verified frontier being Qwen3-235B.

---

The Monolithic Shackles of Hardcoded Geometry

The original codebase was burdened with compile-time constants:

inline constexpr uint32_t D_MODEL    = 2048;
inline constexpr uint32_t VOCAB_SIZE = 102400;
inline constexpr uint32_t GATE_ROWS  = 64;

float* w_q_proj[28]     {};
float* w_input_norm[28] {};

These static numbers were the chains of DeepSeek-16B's specific geometry (d_model 2048, vocab 102400, 64 experts, 28 layers). Running any other model required total recompilation. The system was rigid, blind to the diversity of the frontier.

---

SmoeModelConfig: The Dynamic Contract

We shattered those constants and replaced them with a living runtime configuration struct:

struct SmoeModelConfig {
    uint32_t d_model;                   // hidden dimension
    uint32_t vocab_size;                // embedding table rows
    uint32_t ffn_dim;                   // expert intermediate dim
    uint32_t shared_expert_ffn_dim {0}; // 0 = no shared expert (Qwen3-235B)
    uint32_t num_moe_layers;
    uint32_t max_experts_per_layer;     // routing gate rows
    bool     has_dense_layer_0 {true};  // topology flag (see below)
    bool     has_qk_norm {false};       // Qwen3 per-head Q/K RMS norm
    uint32_t num_heads {16};            // GQA query heads
    uint32_t num_kv_heads {16};         // GQA key/value heads
    uint32_t head_dim {128};
    float    rope_theta {10000.0f};     // RoPE base frequency
    bool     norm_topk_prob {false};    // re-normalise top-k routing weights
    uint32_t moe_top_k {8};             // routed experts per token

    uint32_t gate_rows() const { return max_experts_per_layer; }
};

This configuration awakens automatically at boot time from two sources, each authoritative for what it can actually witness. No clumsy configuration files. No tedious CLI flags.

---

Two Sources of Truth

The vault's arch block carries the mathematical constants that no tensor shape can reveal: rope_theta, moe_top_k, norm_topk_prob, the GQA head geometry, the activation function. The Sculptor serialises them from the checkpoint's HF config.json into the 128-byte SARC block inside the .smoe file (see .smoe Binary Format), and the Scout reads them back at load. The vault is self-describing — copy the .smoe and Scout to any machine and the engine knows the model's lineage.

The tensors themselves remain the authority for structural facts. The Scout parses the safetensors JSON header and probes for well-known keys — what is physically present wins, and any disagreement with the arch block is reported as a warning at startup:

1. "model.embed_tokens.weight"
   → shape[0] = vocab_size
   → shape[1] = d_model

2. "model.layers.0.mlp.gate_proj.weight"
   → EXISTS?  → has_dense_layer_0 = true
               → shape[0] = ffn_dim
   → MISSING? → has_dense_layer_0 = false
               → probe "model.layers.0.mlp.shared_expert.gate_proj.weight"
               → shape[0] = ffn_dim (shared expert intermediate dim)

3. "model.layers.0.mlp.gate.weight" OR
   "model.layers.1.mlp.gate.weight"
   → shape[0] = max_experts_per_layer

---

The has_dense_layer_0 Flag

This is the key that unlocks structural sovereignty:

Model Family	Layer 0 FFN Type	has_dense_layer_0
DeepSeek-MoE-16B	Standard Dense MLP	true
DeepSeek-V2/V3	Standard Dense MLP	true
Qwen2-MoE / Qwen3	MoE from Layer 0	false

When has_dense_layer_0 = false:

• The Layer 0 Dense MLP allocation and execution path is elegantly skipped.
• The MoE routing loop naturally begins at l = 0 instead of l = 1.
• The routing index offset flawlessly shifts: routing[l - moe_start_layer] instead of routing[l - 1].

This single boolean empowers the engine to execute disparate architectural lineages without injecting a single, sluggish conditional branch into the hot path.

---

Shared Expert Key Resolution

DeepSeek and Qwen2-MoE speak different dialects for shared experts:

Family	Key pattern
DeepSeek	model.layers.N.mlp.shared_experts.gate_proj.weight
Qwen2-MoE	model.layers.N.mlp.shared_expert.gate_proj.weight

The Scout seamlessly tries the plural form first. If met with silence, it falls back to the singular form. Architectures with no shared expert at all — Qwen3-235B among them — simply leave shared_expert_ffn_dim = 0 and the shared-expert path is skipped entirely. The user remains undisturbed.

---

Dynamic Buffer Allocation

With fixed-size arrays eradicated, all weight buffers flow freely, allocated at runtime to exactly match their destiny:

// Scales with actual model dimensions — pure efficiency
const size_t EMBED_ELEMS  = (size_t)cfg.vocab_size * cfg.d_model;
const size_t ATTN_W_ELEMS = (size_t)cfg.d_model    * cfg.d_model;

w_embed   = allocate_aligned_float(EMBED_ELEMS);   // 16KB-aligned perfection
w_lm_head = allocate_aligned_float(EMBED_ELEMS);

for (uint32_t l = 0; l < cfg.num_moe_layers + 1; ++l) {
    w_q_proj[l] = allocate_aligned_float(ATTN_W_ELEMS);
    // ...
}

Every allocation commands posix_memalign(16384)—honoring the Apple Silicon page boundary requirement for effortless Metal zero-copy UMA access.

---

Expanding the Frontier

To emancipate a new MoE architecture, the necessary actions are minimal:

1. Sculptor (shatter_moe.py): Inject a regex pattern matching the new model's expert tensor naming convention, and extend read_hf_arch_config if the family uses non-standard config.json keys (unknown model_types fall back to the generic HF key names with a warning).
2. Scout topology detection: Introduce probes for any novel shared expert or routing gate key patterns.
3. has_dense_layer_0 logic: Expand if the model features a radically different pattern.

The C++ engine core—the Streamer, the Metal kernel, the relentless token generation loop—demands zero changes. It is timeless. The only hard boundaries: the kernels implement SiLU/SwiGLU (the arch block's activation field is validated at load), and routing width is capped at 16 experts per token.