attnres_softmax_mix

atlas_core::kimi_k3::attnres

Function attnres_softmax_mix 

Source 
pub fn attnres_softmax_mix(
    sources: &[Vec<f32>],
    query: &[f32],
    norm_w: &[f32],
    eps: f32,
) -> Vec<f32>
Expand description

Softmax mixture over residual sources. sources[0] is conventionally the skip (current partial block). query is [hidden] (*_res_proj).