Struct Glm5NextWeightLoader 

pub struct Glm5NextWeightLoader;

Trait Implementations

impl ModelWeightLoader for Glm5NextWeightLoader

fn binds_vision_encoder(&self) -> bool

This loader binds the tower when — and only when — the operator asked for it with AVAROK_GLM_VISION=1.

It used to answer a flat false, which kept 1.05 GiB/rank of model.visual.* off the GPU entirely. That was not free: measured 2026-08-29, K=3 at 32 K needs 13.58 GiB against 12.07 GiB free, so the tower is exactly the difference between --speculative --num-drafts 2 and a serve that fits. Binding it is a real memory decision, not a correctness cleanup, and a GLM serve that wants the old headroom back has to be given it deliberately.

🪤 This method takes no ModelConfig, so it cannot answer per checkpoint — binds_vision(config) in the server resolves the loader from the config and then asks the loader alone. That is why the gate is an ENV read rather than a config field: it is the one input both this method and parse_glm5_next can see, so the withhold decision and the config.vision decision cannot disagree. Threading the config through the trait would let the gate become a config field; it is a wider change than this one.

The true arm is a real bind (glm5_next_vision.rs), not a load-then-free: factory::build’s reclaim is keyed off whether a tower came back, so it stops firing for this model on its own.

fn load_vision_encoder( &self, store: &WeightStore, config: &ModelConfig, gpu: &dyn GpuBackend, ) -> Result<Option<VisionTower>>

Bind the 347-tensor model.visual.* tower. See crate::weight_loader::glm5_next_vision.

fn defer_predicate(&self, config: &ModelConfig) -> Option<DeferHook>

Keep the MTP block’s full-width routed experts off the device.

nvidia/GLM-5.3-Flash-NVFP4 ships layers.{num_hidden_layers}.mlp. experts.E.{gate,up,down}_proj.weight as BF16 [2048, 4096] against a w4a16-only forward. bind_expert reads each one ONCE and replaces it with an NVFP4 triple ~3.6x smaller, so uploading the originals is pure transient — and on GB10’s unified memory the transient is the problem: measured 2026-09-21 at EP=2, ~7.25 GB per rank resident from the fast loader’s sweep until prune_after_load, i.e. across the whole 45-layer build. That survived a 2048-token boot (host floor 1.74 GB) and killed rank 0 at layer 38 on the 32K qualification shape (935 MB against a 1200 MB floor), BEFORE the MTP bind could free anything.

Deferring them makes the transient structurally impossible: the BF16 stays in the page cache and bind_expert reads it from there.

🪤 None would be wrong ONLY as a silent default — the predicate is dtype-keyed, so returning it on LibertAIDAI/GLM-5.3-Flash-NVFP4 (packed U8 experts everywhere) defers nothing and that checkpoint loads byte-identically. The layer index comes from the config, never a literal: layers.45 is num_hidden_layers, not a property of GLM.

fn supports_tp(&self) -> bool

All three halves shard: DSA by head, KDA by head/channel, the MLP by width (TP) and by expert set (EP).

fn load_mtp_weights( &self, _store: &WeightStore, _config: &ModelConfig, _gpu: &dyn GpuBackend, ) -> Result<Option<MtpWeights>>

MTP is deliberately out of scope for this slice. None = “no speculative head”, which the scheduler already handles; it is not a silent skip of something wired.

fn prune_after_load( &self, store: &mut WeightStore, config: &ModelConfig, gpu: &dyn GpuBackend, ) -> Result<()>

Drop the store’s copy of everything load_layers re-uploaded.

fn load_layers( &self, store: &WeightStore, config: &ModelConfig, gpu: &dyn GpuBackend, _layer_kv_dtypes: &[KvCacheDtype], ) -> Result<Vec<Box<dyn TransformerLayer>>>

Load all transformer layers from the weight store.

fn load_embedding( &self, store: &WeightStore, _config: &ModelConfig, _gpu: &dyn GpuBackend, ) -> Result<DenseWeight>

fn load_final_norm( &self, store: &WeightStore, _config: &ModelConfig, _gpu: &dyn GpuBackend, ) -> Result<DenseWeight>

Load the final RMSNorm weight used before the LM head.

fn load_lm_head( &self, store: &WeightStore, _config: &ModelConfig, _gpu: &dyn GpuBackend, ) -> Result<DenseWeight>

fn precision_schedule(&self, _config: &ModelConfig) -> PrecisionSchedule

Per-(layer, role) weight precision schedule (C.3, 2026-04-25). Default impl returns the empty schedule (every lookup yields Dtype::Inherit), preserving the existing per-checkpoint dtype logic byte-for-byte. Loader-specific implementations can override to honour MODEL.toml’s [precision] block.

fn load_ngram_embedding( &self, _store: &WeightStore, _config: &ModelConfig, _gpu: &dyn GpuBackend, _max_tokens: usize, ) -> Result<Option<NgramEmbedding>>

Build the n-gram embedding, when this architecture fuses hashed n-gram lookups into the input embedding (LongCat / Qwen3.8-Flash-Next).

fn load_mtp_weights_multi( &self, store: &WeightStore, config: &ModelConfig, gpu: &dyn GpuBackend, ) -> Result<Vec<MtpWeights>>

Load MTP weights for multi-module MTP (DeepSeek-V3 / MiniMax-M2 style: N independent transformer modules, each with its own attention + MoE + KV cache). Returns an empty Vec when the checkpoint has no MTP modules, a 1-element Vec for single-module MTP (Qwen3.5 family), or N elements for multi-module.

fn kv_layer_dims(&self, _config: &ModelConfig) -> Vec<(usize, usize)>

Per-layer (num_kv_heads, head_dim) overrides for heterogeneous attention models (e.g. Gemma-4 with sliding 16×256 and full 4×512). Default empty — homogeneous models skip per-layer dims and the KV cache allocator uses the global (num_kv_heads, head_dim). Populated by loaders whose models have different attention geometries per layer. Indexed by attention layer index (same as layer_kv_dtypes).

fn load_dflash_weights( &self, _drafter_store: &WeightStore, _config: &ModelConfig, _gpu: &dyn GpuBackend, _tp_size: usize, ) -> Result<Option<DflashWeights>>

Load DFlash drafter weights from a separate WeightStore pointing at the drafter checkpoint (z-lab/Qwen3.6-{27B,35B-A3B}-DFlash). Default impl returns None so loaders that don’t yet support DFlash silently fall through to the existing MTP path. Override in loaders whose target models pair with a DFlash drafter (Qwen3.5/3.6 family). The same drafter format works across both 27B-dense and 35B-A3B-MoE targets — only the target_hidden_size validated against the drafter’s fc input dimension differs.

fn load_lora_adapters( &self, adapters: &[LoraAdapterInput<'_>], config: &ModelConfig, gpu: &dyn GpuBackend, max_loras: usize, max_lora_rank: usize, ) -> Result<Option<LoraWeights>>

Load one or more startup-static PEFT LoRA adapters from their own WeightStores (the adapter_model.safetensors tensors, already on-device BF16) into the fixed-address rank-padded pool (one slot each).