Struct ExpertLru 

pub struct ExpertLru { /* private fields */ }

Implementations

impl ExpertLru

One byte range of a missed expert as a reader thread sees it: (slot, key, ring slot within the half, byte offset, byte length).

pub fn arena_dev(&self) -> DevicePtr

The arena’s device address (slot i starts at i * layout.bytes).

pub fn drain_slot_changes(&mut self) -> Vec<(u32, u32, i32)>

The (layer, expert, slot | -1) changes since the last call: what a device-side slot table has to learn (assignments and evictions).

pub fn is_device(&self) -> bool

The slots are device memory behind a staging ring.

pub fn fetch_many_on<S: ExpertSource + ?Sized>( &mut self, gpu: &dyn GpuBackend, stream: u64, src: &S, keys: &[(u32, u32)], predicted: &[(u32, u32)], threads: usize, ) -> Result<Vec<ExpertSlot>>

One token’s gather on whichever path this cache was built for: the staging ring (device slots), the reader pool (predicted experts started in the background), or scoped reader threads. The copies of a staged fetch are enqueued on stream ahead of the caller’s launches.

impl ExpertLru

pub fn new( host: *mut u8, dev: DevicePtr, bytes: usize, layout: SlotLayout, ) -> Result<Self>

bytes / layout.bytes slots over the arena at host (device alias dev). Every slot starts empty and at the LRU end.

pub fn n_slots(&self) -> usize

pub fn layout(&self) -> SlotLayout

pub fn resident(&self) -> usize

Experts currently cached.

pub fn contains(&self, layer: u32, expert: u32) -> bool

pub fn stats(&self) -> LruStats

pub fn reset_stats(&mut self)

pub fn begin_token(&mut self)

Open a new epoch: slots fetched from now on are pinned until the next begin_token; everything fetched earlier becomes evictable again.

pub fn slot(&self, i: u32) -> ExpertSlot

pub fn fetch<S: ExpertSource + ?Sized>( &mut self, src: &S, layer: u32, expert: u32, ) -> Result<(ExpertSlot, bool)>

One expert. Returns the slot and whether it was already resident.

pub fn fetch_many<S: ExpertSource + ?Sized>( &mut self, src: &S, keys: &[(u32, u32)], threads: usize, ) -> Result<Vec<ExpertSlot>>

One token’s gather: every key resolved to a slot, misses read on up to threads scoped threads. Duplicate keys are one read. Slots are pinned for the current epoch, so call Self::begin_token first.

impl ExpertLru

pub fn set_evict_random_pct(&mut self, pct: usize)

Strict LRU or a random victim among the oldest pct% (see take_victim).

impl ExpertLru

pub fn set_pool( &mut self, src: Arc<dyn ExpertSource + Send + Sync>, threads: usize, )

Read misses (and predictions) on threads persistent readers from src instead of scoped threads per fetch.

pub fn has_pool(&self) -> bool

pub fn fetch_many_prefetching( &mut self, keys: &[(u32, u32)], predicted: &[(u32, u32)], ) -> Result<Vec<ExpertSlot>>

Slots for keys (misses read urgently), reads of predicted experts started in the background, then the wait for keys only.

pub fn prefetch(&mut self, predicted: &[(u32, u32)]) -> Result<()>

Start reading predicted experts in the background (no wait).

pub fn wait_in_flight(&self)

Block until every read in flight has landed (tests, shutdown).

pub fn set_trace(&mut self, path: &str) -> Result<()>

Append every fetch_many to path as one line: microseconds since the cache was built, the layer, the key count, the misses and the evictions of the call, then the expert ids in request order. The exact access sequence, for replaying cache policies offline.

Trait Implementations

impl Send for ExpertLru