Module device_cache

spark_model::kimi_k3

Module device_cache 

Source
Expand description

GPU-resident hybrid cache. Host HybridCache bytes are prefix snapshot only. Decode must not D2H/H2D conv, recurrent, or the growing MLA KV.

Structs§

DeviceHybridCache
DeviceLayerCache
One layer’s device buffers. MLA k/v are pre-allocated to cap tokens so append is a strided H2D of the new row only.