A cache that grows dynamically as more tokens are generated. This is the default for generative models. It stores the Key and Value states a

Minimal stand-in for a transformers Cache exposing ``key_cache``.

adapt from https://github.com/FFY0/AdaKV.

Base, abstract class for all encoder-decoder caches. Can be used to hold combinations of self-attention and cross-attention caches. Exam

adapt from https://github.com/FFY0/AdaKV.

This class is used to test the LLM Needle Haystack.

kv_head mode: unrepeated K/V at KV_HEADS, query at Q_HEADS.

Old-vs-new torch.equal on outputs AND metadata in the legacy shapes.

Base, abstract class for all caches. The actual data structure is specific to each subclass.

Base class for cache configs

Pure-torch simulation of the per-step decode metadata mutation done by the model forwards (klen_sum += num_cached_heads, max_seqlen_k += 1, cu

adapt from https://github.com/FFY0/AdaKV.

Quantized Cache class that uses `HQQ` as a backend to perform quantization. Current implementation supports `int2`, `int4`, `int8` dtypes. P

Hybrid Cache class to be used with `torch.compile` for Gemma2 models that alternate between a local sliding window attention and global atten

kv_head mode invariants for every in-scope cluster on unrepeated GQA inputs.

Cache for mamba model which does not have attention mechanism and key value states. Arguments: config (`PretrainedConfig):

A drop-in replacement for DynamicCache that conserves GPU memory at the expense of more CPU memory. Useful for generating from models with ve

A quantizer cache similar to what is described in the [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache paper](https://arxiv.org/abs

Configuration class for quantized cache settings. Attributes: backend (`str`, *optional*, defaults to `"quanto"`): Backe

Quantized Cache class that uses `quanto` as a backend to perform quantization. Current implementation supports `int2` and `int4` dtypes only.

A cache that as described in the [Attention Sinks paper](https://arxiv.org/abs/2309.17453). It allows the model to generate beyond the length

Sliding Window Cache class to be used with `torch.compile` for models like Mistral that support sliding window attention. Every time when we

Static Cache class to be used with `torch.compile(model)` and `torch.export()`. Parameters: config (`PretrainedConfig`):