Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch

Quantizes a key/value using a defined quantization method.

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Converts the `DynamicCache` instance into the its equivalent in the legacy cache format. Used for backward compatibility.

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Returns the maximum sequence length of the cached states, if there is any.

Returns the sequence length of the cached states. A layer index can be optionally passed.

Returns the sequence length of the cached states. A layer index can be optionally passed.

Creates K/V cache tensors on a device. Pins memory for CPU tensors. Marks them as static addresses for non-CPU tensors. Args:

Dequantizes back the tensor that was quantized by `self._quantize()`

Performs the actual copy of the layer to device cache.

Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`,

Repeat the cache `repeats` times in the batch dimension. Used in contrastive search.

Only keep the `indices` in the batch dimension of the cache. Used in contrastive search.

Split the current instance into a list of `DynamicCache` by the batch size. This will be used by `_split_model_inputs()` in `generation.utils`

Moves the previous layer cache to the CPU

Reorders the cache for beam search, given the selected beam indices.

Rotates half the hidden dims of the input.

Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch

dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow sca

Prefetch a layer to the device. Needs to be called in order of layer indices.

Crop the past key values up to a new `max_length` in terms of tokens. `max_length` can also be negative to remove `max_length` tokens. This is

Converts a cache in the legacy cache format into an equivalent `DynamicCache`. Used for backward compatibility.

Returns the sequence length of the cached states. A layer index can be optionally passed.

Starts prefetching the next layer cache

Serializes this instance to a Python dictionary. Returns: `Dict[str, Any]`: Dictionary of all the attributes that make up this co

Serializes this instance to a JSON formatted string. Returns: str: JSON formatted string representing the configuration i