kquant_mmvq

Function kquant_mmvq 

Source
pub fn kquant_mmvq(
    gpu: &dyn GpuBackend,
    kernel: KernelHandle,
    w_blocks: DevicePtr,
    y_q8: DevicePtr,
    out_bf16: DevicePtr,
    n: u32,
    k: u32,
    m: u32,
    stream: u64,
) -> Result<()>
Expand description

Decode GEMV on raw blocks: out[m][n] = sum_k xq8[m][k] * W[n][k], m <= 8 (kquant_mmvq_q2_k / kquant_mmvq_q3_k). w_blocks is [n][k/256] raw super-blocks; y_q8 is the block_q8_1 [m][k/32] buffer; out_bf16 is [m][n].