pub struct ModelLevers {Show 66 fields
pub gdn_regresident: bool,
pub gdn_batched_fla: bool,
pub gdn_wy17: bool,
pub gdn_wyn: bool,
pub gemv_sw: bool,
pub decode_ffn_via_gemm: bool,
pub ffn_small_m: bool,
pub holo_moe_down_fp4: bool,
pub holo_moe_gateup_fp4: bool,
pub moe_union_stats: bool,
pub fp32_routing: bool,
pub fp32_gate: bool,
pub frankenstein_decode_via_prefill: bool,
pub k2_diag: bool,
pub decode_split_silu: bool,
pub bf16_tc_prefill: bool,
pub fp8_m64_prefill: bool,
pub int8_prefill: bool,
pub int8_faith5: bool,
pub ffn_nvfp4_mmq: bool,
pub ffn_nvfp4_mmq_down: bool,
pub ffn_mmq: bool,
pub ffn_mmq_down_q4k: bool,
pub fp4_prefill: bool,
pub prefill_v2: bool,
pub moe_grouped_cutlass: bool,
pub moe_grouped_down: bool,
pub moe_prefill_exact_tiles: Option<bool>,
pub moe_prefill_max_load_factor: Option<usize>,
pub moe_prefill_zero: bool,
pub moe_prefill_fp8_down: bool,
pub ssm_w4a4: bool,
pub ssd: bool,
pub ssm_persistent: bool,
pub moe_zero_intermediates: bool,
pub moe_max_m_tiles_estimate: bool,
pub moe_w4a4: bool,
pub shared_w4a4: bool,
pub shared_w4a4_down: bool,
pub dflash_contig_attn: bool,
pub lora_eager: bool,
pub lora_rotate: bool,
pub k4_diag: bool,
pub gemma4_diag: bool,
pub dflash_debug_dump_full: bool,
pub mtp_debug_norms: bool,
pub draft_conf_tau: f32,
pub ssm_save_dump: bool,
pub mla_perseq_fallback: bool,
pub hc_perseq_decode: bool,
pub decode_batch_log: bool,
pub ms_profile: bool,
pub conc_hsd: bool,
pub ep_graphs: bool,
pub gdn_decode_graph: bool,
pub bf16_tc_proj: bool,
pub weight_pre_rotated: bool,
pub ssm_ms_profile: bool,
pub ssm_detail_profile: bool,
pub ssm_gemv_batch4: bool,
pub gdn_fused_conv: bool,
pub moe_legacy_pertoken_decode: bool,
pub max_decode_seqs: u32,
pub shadow_topk: usize,
pub kv_poison: bool,
pub drafter: DrafterContext,
}Expand description
Kernel-path levers for one loaded model.
Plain Copy data resolved from the environment at model construction. Group
membership follows the subsystem the lever steers, so a reader can see at a
glance which part of the forward pass a flag reaches.
Fields§
§gdn_regresident: boolKeep GDN recurrent state in registers across the prefill chunk loop. Default ON (the fold that shipped in PR #369, −7.25 % wall); the env var is an opt-OUT, which is why the field is stored positively and the resolution inverts it.
gdn_batched_fla: boolBatched FLA path for multi-sequence GDN decode.
gdn_wy17: boolWY17 GDN recurrence variant. Ships ON; ATLAS_GDN_WY17=0 opts out.
gdn_wyn: boolWY-N GDN recurrence variant. Ships ON; ATLAS_GDN_WYN=0 opts out.
gemv_sw: boolLossless single-warp decode GEMV (w4a16_gemv_sw, w4a16_gemv_dual_sw).
Ships ON; ATLAS_NO_GEMV_SW=1 restores the 64-thread kernels.
decode_ffn_via_gemm: boolRoute decode FFN through the tile GEMM rather than the scalar GEMV.
ffn_small_m: boolSmall-M FFN GEMM tile shape. Ships ON; ATLAS_FFN_SMALLM=0 opts out.
holo_moe_down_fp4: boolFP4 holo layout for the MoE down projection.
holo_moe_gateup_fp4: boolFP4 holo layout for the MoE gate/up projections.
moe_union_stats: boolCollect per-layer MoE expert-union statistics. Diagnostic.
fp32_routing: boolATLAS_FP32_ROUTING=1 — emit the MoE-input norm in FP32 so the gate
GEMM routes at full precision, removing the bf16-store rounding that
flips experts on gfx1151. Read once per LAYER per DECODE TOKEN from
six call sites via MoeFfnLayer::fp32_routing_active, which also
checks four weight/kernel preconditions — the lever is only the last
term of that conjunction, which is why it lives here and the
preconditions stay on the layer.
fp32_gate: boolATLAS_FP32_GATE=1 — the batched-gate sibling of
Self::fp32_routing.
frankenstein_decode_via_prefill: boolATLAS_FRANKENSTEIN_DECODE_VIA_PREFILL=1 — route the five DFlash
capture layers’ decode through the PREFILL MoE kernel, on the
hypothesis that the decode MoE kernel is the dominant cause of low
drafter acceptance. ~250 us per capture layer, so ~1.25 ms/token
against a ~58 ms/token decode. Diagnostic; non-capture layers are
untouched.
k2_diag: boolATLAS_K2_DIAG=1 — K=2 routed-decode diagnostics.
decode_split_silu: boolSplit SiLU+down on the decode path: silu_mul into gate_out, then a
separate w4a16_decode_gemv for down. Ships ON;
ATLAS_NO_DECODE_SPLIT_SILU (presence) restores the fused kernel.
A LoRA adapter pins this path on regardless — the fused alternative
never materialises silu(gate)*up, which the down delta must
contract over — so the call site is levers.decode_split_silu || self.lora.is_some().
bf16_tc_prefill: boolATLAS_BF16_TC_PREFILL (presence) — BF16 tensor-core prefill GEMM.
Read here only; the usable gate is derived at the call site AFTER
v1/v2 selection, from the handle actually launched. Gating on v1’s
handle while dispatching v2 admitted launches of a kernel the target
may not carry.
fp8_m64_prefill: boolATLAS_FP8_M64_PREFILL (presence) — m16n8k32 e4m3 M64 prefill GEMM,
~1.47x vs v2 BF16. Lossy (cosine 0.9997), so opt-in only.
int8_prefill: boolATLAS_INT8_PREFILL (presence) — requant→int8_gemm_faith2 prefill
(cosine 0.999978 vs the host full-precision dequant GEMM).
int8_faith5: boolATLAS_INT8_FAITH5 (presence) — int32 per-sub-block accumulation,
which breaks the MMA→scale dependency chain. Same kernel signature
and launch geometry as faith2, so it is a handle swap.
ffn_nvfp4_mmq: boolVendored llama NVFP4 W4A4 MMQ for the gate/up prefill GEMMs
(~80 TFLOP/s vs t_m128’s ~51). Ships ON;
ATLAS_NO_FFN_NVFP4_MMQ (presence) is the kill switch.
ffn_nvfp4_mmq_down: boolThe same MMQ arm for the down projection — t_m128 runs the narrow-N
down at only ~34 TFLOP/s. Ships ON; ATLAS_NO_FFN_NVFP4_MMQ_DOWN
(presence) is the kill switch. Separate from
Self::ffn_nvfp4_mmq because down is the heavy-tailed projection
(W4A4 cosine 0.9961) and gets its own gate.
ffn_mmq: boolATLAS_FFN_MMQ (presence) — Q4_K MMQ prefill arm.
ffn_mmq_down_q4k: boolATLAS_FFN_MMQ_DOWN_Q4K (presence) — keep the down projection ON
Q4_K instead of the near-lossless faith2 NVFP4 hybrid.
Stores the POSITIVE of a variable whose call site reads the negative
(!levers.ffn_mmq_down_q4k), the same shape as
Self::moe_legacy_pertoken_decode. down = SiLU(gate)*up is
heavy-tailed and Q4_K superblock scaling clips it — BFCL multiple
−4.0%, which is why llama promotes only down→Q6_K.
fp4_prefill: boolATLAS_FP4_PREFILL (presence) — native W4A4 FP4 tensor cores
(sm_121a), NVFP4 weights used directly with no requant. Lossy
(cos ~0.99 vs fp32).
prefill_v2: boolThe v2 BF16 t_m128 prefill kernel — faster and bit-identical to v1.
Ships ON; ATLAS_DISABLE_PREFILL_V2 (presence) forces v1 so the two
can be compared for TTFT in one binary.
moe_grouped_cutlass: boolATLAS_HOLO_MOE_GROUPED_CUTLASS=1 — single-launch CUTLASS grouped
NVFP4 gate_up. Off by default; unset falls back to the hand-rolled
fused FP4/FP8 grouped kernels.
moe_grouped_down: boolATLAS_HOLO_MOE_GROUPED_DOWN=1 — take the down projection through
the same CUTLASS grouped path. Requires
Self::moe_grouped_cutlass; a separate gate because down consumes
the already-expert-contiguous post-SiLU output and needs no gather.
moe_prefill_exact_tiles: Option<bool>ATLAS_MOE_PREFILL_EXACT_TILES=1|0 overrides the tile bound;
None (unset) defers to the checkpoint — the win was measured on
NVFP4, so the default is scoped to where it was measured.
Tri-state on purpose. Measured: exact_tiles ON gave p90 +4.9% against a +5.0% limit (0.1% from failing the gate) and OFF gave p90 −5.0%, while the median barely moved either way (+0.1% vs −0.9%). Only the tail shows it, so both directions must stay reachable. Graph capture forces it off regardless — the bound is read back from device memory.
moe_prefill_max_load_factor: Option<usize>ATLAS_MOE_PREFILL_MAX_LOAD_FACTOR=<n> — cap the per-expert tile
bound at n times the average when exact tiles are off. None (unset
or 0) means the worst case.
moe_prefill_zero: boolATLAS_MOE_PREFILL_ZERO=1 — memset the grouped scratch before
dispatch. Implied by EP (ctx.comm.is_some()). In non-EP the sort
produces a dense permutation over exactly the rows the grouped
kernels write, so skipping the clear removes ~138 MB/layer on Holo.
moe_prefill_fp8_down: boolATLAS_MOE_PREFILL_FP8_DOWN=1 — FP8 grouped GEMM for the routed
down projection.
ssm_w4a4: boolW4A4 native-FP4 SSM projections at N >= 512. Ships ON;
ATLAS_NO_SSM_W4A4 (presence) is the kill switch.
ssd: boolThe chunked SSD scan. Ships ON; ATLAS_NO_SSD (presence) falls back
to the sequential scan. Gated additionally on ssd_scan_fits, since
Nano-30B’s state_size=128 overflows the shared-memory budget that
Puzzle-75B’s 96 fits — which is why that never surfaced until it did.
ssm_persistent: boolThe persistent SSM prefill kernel, which keeps H in shared memory and
is only reachable when SSD is unavailable. Ships ON;
ATLAS_NO_SSM_PERSISTENT (presence) disables, for a same-binary A/B
against the sequential scan.
moe_zero_intermediates: boolZero the grouped-MoE intermediate arena buffers before dispatch.
Ships ON; ATLAS_MOE_NO_ZERO_INTERMEDIATES (presence) skips.
Defence in depth: these buffers are reused across requests and nothing else clears them, so a row a future change fails to write would leak the PREVIOUS request’s activations rather than merely being wrong. Asked twice in one call before this — once for up, once for down.
moe_max_m_tiles_estimate: boolATLAS_MOE_MAX_M_TILES_ESTIMATE (presence) — restore the old
average-based tile bound. A/B only; the comment at the site says it
is NOT safe to serve on, because the estimate can under-bound the
worst case of one expert taking every routed token.
moe_w4a4: boolATLAS_MOE_W4A4 (presence) — W4A4 grouped up-projection at N >= 512.
W4A4 for the shared-expert UP projection at N >= 512. Ships ON;
ATLAS_NO_SHARED_W4A4 (presence) is the kill switch.
ATLAS_SHARED_W4A4_DOWN (presence) — the DOWN half of the same, and
a SEPARATE opt-in: down is the heavy-tailed projection, so it does not
inherit Self::shared_w4a4.
dflash_contig_attn: boolContiguous-attention path for the DFlash head.
lora_eager: boolApply LoRA eagerly at load instead of at each forward.
lora_rotate: boolAllow hot rotation of LoRA adapters.
k4_diag: boolK=4 chain-widening diagnostics.
gemma4_diag: boolPer-layer hidden-state norm dumps on the Gemma-4 decode path. Heavy — one device-to-host copy per layer.
dflash_debug_dump_full: boolATLAS_DFLASH_DEBUG_DUMP_FULL=1 — the model-side half of the DFlash
full dump: emit the whole token sequence ONCE so a Python reference
can run the same tokens through HF transformers.
★ The SAME variable that crate::layers::dflash_head::levers::DFlashLevers::debug_dump_full
carries. Two structs, one flag — deliberately, because the two halves
of the dump are armed together by design and the head is not reachable
from TransformerModel (proposer is a dyn DraftProposer, so there
is nothing to read the head’s levers through without a downcast).
the_two_halves_of_the_dflash_dump_agree pins that the two
resolutions cannot drift, which is what makes the duplication safe —
an unchecked second spelling of one lever is how
ATLAS_DSPARK_ANCHOR_BIAS came to have two implementations.
mtp_debug_norms: boolATLAS_MTP_DEBUG_NORMS=1 — per-stage norm dumps inside the MTP
drafter’s forward_one, which asked for it FOUR times per drafted
token, each read only to decide whether to do nothing.
draft_conf_tau: f32ATLAS_MTP_DRAFT_CONF=<t> — confidence floor for submitting drafts
to verification, clamped to [0.0, 0.99]. 0.0 (unset) disables.
When the drafter’s chain confidence (the min top-1 softmax prob
across one propose’s drafts) is below this, the drafts are discarded
and the next step decodes serially, skipping a verify that would most
likely reject. Economics at K=1 on the 35B MoE: verify ~35 ms for
1+accepted tokens against decode+propose 21 ms for 1, so a draft is
only worth verifying at p(accept) > 0.66. STAGED OFF pending its
measured A/B.
Three of its four readers asked per propose whether the feature was
on, i.e. paid the environment lock to learn it was off. The fourth,
MtpHead::last_confidence, is reached only when it is already ON, so
it keeps its own read and its own contract — see the note there.
ssm_save_dump: boolATLAS_SSM_SAVE_DUMP (presence) — the CBD scratch/SSM-state
fingerprint probe. Asked THREE times per decode step by the decode
path alone, each read only to decide whether to do nothing.
mla_perseq_fallback: boolATLAS_MLA_PERSEQ_FALLBACK=1|true — route MLA batches through the
per-sequence path instead of the batched one.
hc_perseq_decode: boolATLAS_HC_PERSEQ_DECODE=1 — per-sequence hyper-connection decode.
ORed with qsa_active, and the routing decision is resolved ABOVE
the EP branch on purpose: it used to sit below, so under EP a
QSA-active batch returned before reaching the gate, landed on the
batched multi-seq path, and died on its guard.
decode_batch_log: boolATLAS_DECODE_BATCH_LOG=1 — log the batch’s slot/position vectors
each step.
ms_profile: boolATLAS_MS_PROFILE=1 — per-phase multi-seq profiling, which forces
eager execution so the per-phase syncs are legal under capture.
NOT Self::ssm_ms_profile, which is ATLAS_SSM_MS_PROFILE. Two
different variables one underscore apart, both live.
conc_hsd: boolATLAS_CONC_HSD=1|true — per-sequence hidden-state dump, to localize
where pos >= 1 diverges from pos 0 in concurrent batched decode.
ep_graphs: boolATLAS_EP_GRAPHS=1|true — allow CUDA-graph capture under expert
parallelism. The EP all-reduce queues ncclSend/Recv plus a local add
on the capture stream and NCCL >= 2.9 supports capture, so this MAY
capture cleanly; env-gated so a deploy can revert instantly if
capture crashes or replay hangs.
gdn_decode_graph: boolATLAS_GDN_DECODE_GRAPH=1|true — capture the whole single-token GDN
HeadParallel TP decode forward (~130 kernels plus the per-layer TP
all-reduces) into one replayable graph. Default OFF.
bf16_tc_proj: boolBF16 tensor-core attention projections: dequant FP4 to BF16 and use a BF16 MMA instead of the default path, which crushes activations to FP8 E4M3. Removes the FP8 prefill perturbation on those projections.
weight_pre_rotated: boolThe checkpoint’s attention weights are ALREADY Hadamard-rotated at load
(TQ_PLUS_WEIGHT_ROTATION), so the runtime must not rotate again.
A property of the loaded checkpoint, and the SSOT for it. It previously
had FIVE implementations of the same =1-or-true test — four raw
std::env::var calls on attention paths (one per attention layer per
DECODE TOKEN in decode/attention_forward.rs, one per layer per
batched decode step in multi_seq/attn.rs, two per layer per prefill
chunk) plus a fifth in the weight loader, whose #[allow(dead_code)]
was stale — attention_arms.rs calls it. Reading this per token cost
an allocation and the process-wide environment lock on the hottest path
in the model, and five copies of one predicate is how a flag ends up
decoded two different ways in one binary.
ssm_ms_profile: boolPer-step multi-sequence SSM profiling dump.
ssm_detail_profile: boolFiner per-sub-step SSM profiling inside the batched recurrence.
ssm_gemv_batch4: boolShips ON: use the batch-4 GEMV tier for the SSM projections when the
kernel is resolved and n <= 16. ATLAS_SSM_GEMV_BATCH4=0 opts out.
gdn_fused_conv: boolFuse the GDN conv with the F32 norm when the head geometry allows.
moe_legacy_pertoken_decode: boolTake the pre-token-major MoE decode kernel. The field stores the
POSITIVE of the variable’s name, so the call site reads
!levers.moe_legacy_pertoken_decode for the default token-major path —
the inversion lives here, once, rather than at the branch.
max_decode_seqs: u32Configured max decode batch (--max-batch-size), the reference count
the split-K attention split count is pinned to. Not from the
environment: TransformerModel::new writes it from the serve arg.
It pins DETERMINISM — the online-softmax split-merge is
non-associative, so a sequence decoded alone must see the same
reduction tree as one co-batched with fifteen others. Held in a
OnceLock it was also idempotent, so a second model with a different
max batch would silently keep the first model’s split count.
shadow_topk: usizeATLAS_MTP_SHADOW_TOPK=k (0 = off, clamped to 8): the drafter D2Hs
its logits and logs the top-k candidates. Observational only.
kv_poison: boolATLAS_KV_POISON=1 — fill a fresh KV block with NaN instead of zero,
the discriminator for the “unwritten fresh tail block read”
hypothesis. A diagnostic that changes what the kernels READ, so it
must not leak across a swap.
drafter: DrafterContextMTP drafter context policy (ATLAS_NO_DRAFTER_CONTEXT /
ATLAS_DRAFTER_PREFILL_ONLY), resolved and logged once per model.
The two halves are coupled — prefill without carry is a measured
−927 ms/turn loss — so they travel as one value.
Implementations§
Source§impl ModelLevers
impl ModelLevers
Sourcepub fn get() -> &'static Self
pub fn get() -> &'static Self
The process-wide levers, resolved from the environment EXACTLY ONCE.
★ USE THIS, NOT Self::from_env. Every field here is a pure function
of ATLAS_* environment variables, which cannot change after start —
the runtime set_var that could have changed them was deliberately
removed. So this is a process constant and must be computed once.
It was not. from_env reads ~30 environment variables, each allocating
a String, and three call sites invoked it from hot paths.
MEASURED: 32,513 resolutions in a single concurrency-sweep — which
matches 48 layers x ~680 prefills, i.e. once per layer per prefill from
qwen3_attention::prefill_weights. Each of those also re-ran
drafter_context::resolve_from_env and its logging.
Returns a reference so callers cannot accidentally keep re-resolving;
ModelLevers is Copy, so *ModelLevers::get() is free when an owned
value is wanted.
Sourcepub fn from_env() -> Self
pub fn from_env() -> Self
Resolve from the environment, unconditionally.
Prefer Self::get. This exists for the one caller that needs an OWNED,
MUTABLE copy — the model build overwrites max_decode_seqs with the
batch size — and for tests that want a fresh read. Calling it in a hot
path re-reads every ATLAS_* variable.
Trait Implementations§
Source§impl Clone for ModelLevers
impl Clone for ModelLevers
Source§fn clone(&self) -> ModelLevers
fn clone(&self) -> ModelLevers
1.0.0 · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read more