Functions — github.com/Zefan-Cai/KVCache-Factory

hub / github.com/Zefan-Cai/KVCache-Factory / functions

Functions410 in github.com/Zefan-Cai/KVCache-Factory

⨍ Functions 410◇ Types & classes 47

Method__init__

(self)

tests/test_generation_state.py:14

Method__iter__

(self)

pyramidkv/pyramidkv_utils.py:55

Method__iter__

allows `dict(obj)` for situations where obj may be a dict or QuantizationConfigMixin

pyramidkv/cache_utils_think.py:156

Method__iter__

Support for backwards-compatible `past_key_value` iteration, e.g. `for x in past_key_value:` to iterate over keys and values

pyramidkv/cache_utils_think.py:338

Method__len__

(self)

pyramidkv/pyramidkv_utils.py:52

Method__len__

Support for backwards-compatible `past_key_value` length, e.g. `len(past_key_value)`. This value corresponds to the number of layers

pyramidkv/cache_utils_think.py:346

Method__len__

Support for backwards-compatible `past_key_value` length, e.g. `len(past_key_value)`. This value corresponds to the number of layers

pyramidkv/cache_utils_think.py:1318

Method__repr__

(self)

pyramidkv/cache_utils_think.py:162

Method_dequantize

(self, qtensor)

pyramidkv/cache_utils_think.py:762

Method_dequantize

(self, qtensor)

pyramidkv/cache_utils_think.py:821

Function_flash_attention_forward

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, then compu

pyramidkv/llama_model_think.py:33

Method_quantize

(self, tensor, axis)

pyramidkv/cache_utils_think.py:757

Method_quantize

(self, tensor, axis)

pyramidkv/cache_utils_think.py:808

Method_sliding_update

(self, cache_position, layer_idx, key_states, value_states, k_out, v_out, max_cache_len)

pyramidkv/cache_utils_think.py:1516

Method_static_update

(self, cache_position, layer_idx, key_states, value_states, k_out, v_out, max_cache_len)

pyramidkv/cache_utils_think.py:1544

Functionadaptive_LlamaModel_forward

( self, input_ids: torch.LongTensor = None, attention_mask: Optional[torch.Tensor] = N

pyramidkv/llama_model.py:2802

Functionadaptive_LlamaModel_forward

( self, input_ids: torch.LongTensor = None, attention_mask: Optional[torch.Tensor] = N

pyramidkv/llama_model_think.py:432

Functionadaptive_MistralModel_forward

( self, input_ids: torch.LongTensor = None, attention_mask: Optional[torch.Tensor] = None, pos

pyramidkv/mistral_model.py:3044

Methodbatch_repeat_interleave

Repeat the cache `repeats` times in the batch dimension. Used in contrastive search.

pyramidkv/cache_utils_think.py:1424

Methodbatch_select_indices

Only keep the `indices` in the batch dimension of the cache. Used in contrastive search.

pyramidkv/cache_utils_think.py:1430

Methodbatch_split

Split the current instance into a list of `DynamicCache` by the batch size. This will be used by `_split_model_inputs()` in `generation.utils`

pyramidkv/cache_utils_think.py:1396

Functionclassification_score

(prediction, ground_truth, **kwargs)

metrics.py:89

Functioncode_sim_score

(prediction, ground_truth, **kwargs)

metrics.py:80

Functioncount_score

(prediction, ground_truth, **kwargs)

metrics.py:47

Methodcrop

Crop the past key values up to a new `maximum_length` in terms of tokens. `maximum_length` can also be negative to remove `maximum_length` tok

pyramidkv/cache_utils_think.py:1390

Methodfrom_batch_splits

This is the opposite of the above `batch_split()` method. This will be used by `stack_model_outputs` in `generation.utils`

pyramidkv/cache_utils_think.py:484

Methodfrom_batch_splits

This is the opposite of the above `batch_split()` method. This will be used by `stack_model_outputs` in `generation.utils`

pyramidkv/cache_utils_think.py:1409

Methodfrom_dict

Constructs a CacheConfig instance from a dictionary of parameters. Args: config_dict (Dict[str, Any]): Dictionary contain

pyramidkv/cache_utils_think.py:109

Methodfrom_legacy_cache

Converts a cache in the legacy cache format into an equivalent `DynamicCache`.

pyramidkv/pyramidkv_utils.py:108

Methodfrom_legacy_cache

Converts a cache in the legacy cache format into an equivalent `EncoderDecoderCache`.

pyramidkv/cache_utils_think.py:1338

Functionget_cuda_bare_metal_version

pyramidkv/pyramidkv_utils.py:97

Methodget_max_length

Returns the maximum sequence length of the cached states. DynamicCache does not have a maximum length.

pyramidkv/cache_utils_think.py:433

Methodget_max_length

Returns the maximum sequence length of the cached states.

pyramidkv/cache_utils_think.py:912

Methodget_max_length

Returns the maximum sequence length of the cached states.

pyramidkv/cache_utils_think.py:1137

Methodget_max_length

(self)

pyramidkv/cache_utils_think.py:1258

Methodget_max_length

(self)

pyramidkv/cache_utils_think.py:1580

Methodget_results

(self)

run_needle_in_haystack.py:484

Methodget_seq_length

(self, layer_idx: Optional[int] = 0)

pyramidkv/pyramidkv_utils.py:90

Methodget_seq_length

Returns the sequence length of the cached states. A layer index can be optionally passed.

pyramidkv/cache_utils_think.py:689

Methodget_seq_length

Returns the sequence length of the cached states that were seen by the model.

pyramidkv/cache_utils_think.py:1130

Methodget_seq_length

Returns the sequence length of the cached states. A layer index can be optionally passed.

pyramidkv/cache_utils_think.py:1353

Methodget_seq_length

(self, layer_idx: Optional[int] = 0)

pyramidkv/cache_utils_think.py:1585

Functioninfer_attention

Retrieve attention weights and return them as a list of tensors. Args: prompt (str): The input text to analyze. amp (flo

pyramidkv/viztools/infer.py:4

Methodinit_metadata

(num_heads, k_lens, klen_sum, max_seqlen_k)

pyramidkv/pyramidkv_utils.py:1016

Methodinit_metadata

(num_heads, k_lens, klen_sum, max_seqlen_k)

pyramidkv/pyramidkv_utils.py:1161

Functioninit_think

(self)

pyramidkv/pyramidkv_utils.py:1290

Functionllama_attn_forward_CAM

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/llama_model.py:848

Functionllama_attn_forward_H2O

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/llama_model.py:1210

Functionllama_attn_forward_L2Norm

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/llama_model.py:474

Functionllama_attn_forward_PyramidKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/llama_model.py:102

Functionllama_attn_forward_SnapKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/llama_model.py:1933

Functionllama_attn_forward_SnapKV_ThinK

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/llama_model_think.py:101

Functionllama_attn_forward_StreamingLLM

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/llama_model.py:1572

Functionllama_flash_attn2_forward_AdaKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.LongTensor] = None, positi

pyramidkv/llama_model.py:2294

Functionllama_flash_attn2_forward_CAM

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.LongTensor] = None, positi

pyramidkv/llama_model.py:1090

Functionllama_flash_attn2_forward_H2O

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.LongTensor] = None, positi

pyramidkv/llama_model.py:1452

Functionllama_flash_attn2_forward_HeadKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.LongTensor] = None, positi

pyramidkv/llama_model.py:2447

Functionllama_flash_attn2_forward_L2Norm

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.LongTensor] = None, positi

pyramidkv/llama_model.py:714

Functionllama_flash_attn2_forward_PyramidKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.LongTensor] = None, positi

pyramidkv/llama_model.py:342

Functionllama_flash_attn2_forward_SnapKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.LongTensor] = None, positi

pyramidkv/llama_model.py:2175

Functionllama_flash_attn2_forward_StreamingLLM

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.LongTensor] = None, positi

pyramidkv/llama_model.py:1813

Functionllama_sdpa_attn_forward_CAM

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/llama_model.py:972

Functionllama_sdpa_attn_forward_H2O

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/llama_model.py:1334

Functionllama_sdpa_attn_forward_L2Norm

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/llama_model.py:597

Functionllama_sdpa_attn_forward_PyramidKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/llama_model.py:225

Functionllama_sdpa_attn_forward_SnapKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/llama_model.py:2057

Functionllama_sdpa_attn_forward_StreamingLLM

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/llama_model.py:1696

Functionminference_attn_forward

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/minference.py:14

Functionmistral_attn_forward_CAM

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:973

Functionmistral_attn_forward_H2O

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:111

Functionmistral_attn_forward_L2Norm

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:541

Functionmistral_attn_forward_PyramidKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:1835

Functionmistral_attn_forward_SnapKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:2265

Functionmistral_attn_forward_StreamingLLM

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:1403

Functionmistral_flash_attn2_forward_AdaKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:2691

Functionmistral_flash_attn2_forward_CAM

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:1233

Functionmistral_flash_attn2_forward_H2O

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:371

Functionmistral_flash_attn2_forward_HeadKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:2866

Functionmistral_flash_attn2_forward_L2Norm

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:800

Functionmistral_flash_attn2_forward_PyramidKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:2095

Functionmistral_flash_attn2_forward_SnapKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:2522

Functionmistral_flash_attn2_forward_StreamingLLM

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:1665

Functionmistral_sdpa_attn_forward_CAM

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:1086

Functionmistral_sdpa_attn_forward_H2O

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:224

Functionmistral_sdpa_attn_forward_L2Norm

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:654

Functionmistral_sdpa_attn_forward_PyramidKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:1948

Functionmistral_sdpa_attn_forward_SnapKV

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:2375

Functionmistral_sdpa_attn_forward_StreamingLLM

( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_i

pyramidkv/mistral_model.py:1518

Functionplot_attention_heatmap

Visualizes attention weights as a heatmap. Args: attentions (torch.Tensor): Attention weights tensor of shape (num_heads, seq_len, s

pyramidkv/viztools/visualization.py:8

Functionprepare_inputs_for_generation_llama

( self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs )

pyramidkv/llama_model.py:2733

Functionprepare_inputs_for_generation_llama

( self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs )

pyramidkv/llama_model_think.py:363

Functionprepare_inputs_for_generation_llama_new

( self, input_ids, past_key_values=None, attention_mask=None, inputs_e

pyramidkv/llama_model.py:2656

Functionprepare_inputs_for_generation_llama_new

( self, input_ids, past_key_values=None, attention_mask=None, inputs_e

pyramidkv/llama_model_think.py:286

Functionprepare_inputs_for_generation_mistral

( self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs )

pyramidkv/mistral_model.py:3216

Functionprepare_inputs_for_generation_mistral_new

( self, input_ids, past_key_values=None, attention_mask=None, inputs_e

pyramidkv/mistral_model.py:3164

Functionqa_f1_score

(prediction, ground_truth, **kwargs)

metrics.py:128

Functionqa_f1_zh_score

(prediction, ground_truth, **kwargs)

metrics.py:137

Methodreorder_cache

Saves the beam indices and reorders the cache when the tensor is back to its device.

pyramidkv/cache_utils_think.py:567

Methodreorder_cache

Reorders the cache for beam search, given the selected beam indices.

pyramidkv/cache_utils_think.py:1374

← previous next →201–300 of 410, ranked by callers