Module attnres 

Block AttnRes CPU reference.

Each sublayer mixes a learned softmax over completed block residuals plus the current intra-block partial sum (Moonshot Block AttnRes):

K_i = RMSNorm(V_i)
α   = softmax_i( q · K_i )
h   = Σ α_i V_i

q is the per-layer *_res_proj row. Mix=0 is the identity skip (return the partial / skip source). Mix=1 is the full softmax mixture.

Structs

AttnResHub

Stream map keyed by hidden/residual pointer bits (DevicePtr.0). Layer 0 inserts; last layer removes on success. Any Err drops the entry (umbrella 940bd4eeb).

Functions

attnres_blend

Test / ablation lever: mix=0 returns skip; mix=1 returns mixed.

attnres_mix

Apply AttnRes with an explicit mix lever. mix=0 is identity skip.

attnres_softmax_mix

Softmax mixture over residual sources. sources[0] is conventionally the skip (current partial block). query is [hidden] (*_res_proj).

rms_norm

Vanilla RMSNorm: x * w / sqrt(mean(x^2) + eps).