Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`,

Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`,

Replicate PyramidKVCluster's per-layer kept-token schedule (long branch).

kv_head-granularity path (groups > 1: unrepeated GQA key/value tensors). Window scores are computed with transiently repeated keys, group-red

kv_head-granularity path (groups > 1: unrepeated GQA key/value tensors). Window scores are computed with transiently repeated keys, group-red

kv_head-granularity path (groups > 1: unrepeated GQA key/value tensors). Window scores are computed with transiently repeated keys, then the

kv_head-granularity path (groups > 1: unrepeated GQA key/value tensors). H2O keeps its full-matrix scoring; the only extra full-size tensor i

Compress one adjacent-layer KV pair using the MiniCache core contract.

Crop the past key values up to a new `max_length` in terms of tokens. `max_length` can also be negative to remove `max_length` tokens. This is

Returns the sequence length of the cached states. A layer index can be optionally passed.

Starts prefetching the next layer cache

Restore approximate current/previous states from a MiniCache pair.

Checks to see if a result has already been evaluated or not

evaluation metric for RULER preds: List[str] refs: List[List[str]]

Serializes this instance to a Python dictionary. Returns: `Dict[str, Any]`: Dictionary of all the attributes that make up this co

Serializes this instance to a JSON formatted string. Returns: str: JSON formatted string representing the configuration i

Update persistent token importance with optional exponential decay.

Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch

Support for backwards-compatible `past_key_value` indexing, e.g. `past_key_value[0][0].shape[2]` to get the sequence length.

Gets the cache for this layer to the device. Prefetches the next and evicts the previous layer.

Support for backwards-compatible `past_key_value` indexing, e.g. `past_key_value[0][0].shape[2]` to get the sequence length.

:param needle: The needle to be found in the haystack. Default is None. :param haystack_dir: The directory of text files to u