Struct SharedV41 

pub struct SharedV41 {
    pub compress_kv: Option<DevicePtr>,
    pub compress_len: usize,
    pub index_k: Option<DevicePtr>,
    pub topk_idxs: Vec<i32>,
    pub topk: usize,
    pub candidates: Vec<bool>,
    pub cand_width: usize,
}

The SharedAttentionRuntime: what source layers publish for the layers below them.

Fields

compress_kv: Option<DevicePtr>

the current kv source’s latent cache and how many groups it holds

compress_len: usize

index_k: Option<DevicePtr>

the current kv source’s index-key cache

topk_idxs: Vec<i32>

[queries][topk], offset for the window rows, -1 = absent

topk: usize

candidates: Vec<bool>

[queries][cand_width]

cand_width: usize

Trait Implementations

impl Default for SharedV41

fn default() -> SharedV41

Returns the “default value” for a type.