bounded_gate

Function bounded_gate 

Source
pub fn bounded_gate(
    z: &[f32],
    dt_bias: &[f32],
    a_log: &[f32],
    heads: usize,
    head_dim: usize,
    lower_bound: Option<f32>,
) -> Vec<f32>
Expand description

KDA forget-gate in log space (FLA use_gate_in_kernel).

  • Some(lb): lb * sigmoid(exp(A_log) * (z + dt_bias)) (production -5)
  • None: -exp(A_log) * softplus(z + dt_bias) (0.40B twin / FLA default)