kquant_mmvq_experts

Function kquant_mmvq_experts 

Source
pub fn kquant_mmvq_experts(
    gpu: &dyn GpuBackend,
    kernel: KernelHandle,
    vxs: DevicePtr,
    y_q8: DevicePtr,
    out_bf16: DevicePtr,
    n: u32,
    k: u32,
    m: u32,
    n_experts: u32,
    y_stride_bytes: u32,
    stream: u64,
) -> Result<()>
Expand description

Prefill GEMM on raw blocks through the vendored MMQ engine: out[m][n] (bf16) = A_q8[m][k] x W[n][k]. a_q8 must be in the type’s MMQ q8_1 layout (D2S6 for Q2_K, D4 for Q3_K); smem is the matching *_MMQ_SMEM. kquant_mmvq_*_experts: one launch over n_experts experts whose block pointers sit in the device table vxs (n_experts pointers). The activation for expert e is y_q8 + e * y_stride_bytes (0 = shared), its output out_bf16 + e * m * n. Same per-row math as kquant_mmvq.