attnres_softmax_mix

spark_model::kimi_k3

Function attnres_softmax_mix 

Source 
pub fn attnres_softmax_mix(
    sources: &[Vec<f32>],
    query: &[f32],
    norm_w: &[f32],
    eps: f32,
) -> Vec<f32>
Expand description

Softmax mixture over residual sources. sources[0] is conventionally the skip (current partial block). query is [hidden] (*_res_proj).