MCPcopy Create free account

hub / github.com/Jingyu6/speculative_prefill / functions

Functions156 in github.com/Jingyu6/speculative_prefill

⨍ Functions 156◇ Types & classes 26↳ Endpoints 2
↓ 8 callersFunction_num_cpu_cores
()
eval/lm_eval_patch/scrolls/task.py:98
↓ 7 callersFunctionenable_prefill_spec
( spec_model: str = 'meta-llama/Llama-3.2-1B-Instruct', spec_config_path: Optional[str] = None )
speculative_prefill/vllm_patch/__init__.py:43
↓ 7 callersMethodgenerate
( self, context: str, input: str, prompt_format: str, dataset_name
rag_baseline/rag_model.py:47
↓ 5 callersFunction_process_doc_prepended_question
(doc)
eval/lm_eval_patch/scrolls/task.py:61
↓ 3 callersFunctionget_spec_config
()
speculative_prefill/vllm_patch/config.py:61
↓ 3 callersFunctionrun_to_completion
(profile_dir: Optional[str] = None)
eval/minference_latency.py:53
↓ 3 callersFunctionrun_to_completion
(profile_dir: Optional[str] = None)
speculative_prefill/vllm_benchmarks/latency.py:45
↓ 2 callersMethod_get_model_num_layers
(self)
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:371
↓ 2 callersMethod_normalize_answer
(text)
eval/lm_eval_patch/scrolls/task.py:365
↓ 2 callersMethod_prepare_query_buffer
(self)
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:80
↓ 2 callersMethod_run_non_driver_rank
(self)
speculative_prefill/vllm_patch/worker/spec_prefill_worker.py:168
↓ 2 callersMethod_scrolls_metrics
(self)
eval/lm_eval_patch/scrolls/task.py:207
↓ 2 callersMethoddownload
(self, *args, **kwargs)
eval/lm_eval_patch/scrolls/task.py:160
↓ 2 callersMethodexecute_model
( self, execute_model_req: ExecuteModelRequest | None = None )
speculative_prefill/vllm_patch/worker/spec_prefill_worker.py:137
↓ 2 callersFunctionf1_score
(prediction, ground_truth, **kwargs)
eval/long_bench/metrics.py:118
↓ 2 callersFunctionget_attn_scores
(model_name)
visualization/attn_correlation.py:58
↓ 2 callersMethodget_cache_block_size_bytes
(self)
speculative_prefill/vllm_patch/worker/spec_prefill_worker.py:204
↓ 2 callersMethodload_model
(self, *args, **kwargs)
speculative_prefill/vllm_patch/worker/spec_prefill_worker.py:105
↓ 2 callersFunctionlower
(text)
eval/long_bench/metrics.py:25
↓ 2 callersFunctionmain
(args)
eval/qps_client.py:104
↓ 2 callersFunctionmain
(args: argparse.Namespace)
speculative_prefill/vllm_benchmarks/latency.py:21
↓ 2 callersFunctionnormalize_answer
Lower text and remove punctuation, articles and extra whitespace.
eval/long_bench/metrics.py:12
↓ 2 callersFunctionnormalize_zh_answer
Lower text and remove punctuation, extra whitespace.
eval/long_bench/metrics.py:31
↓ 2 callersFunctionpatch_vllm_tp
()
eval/long_bench/minference_vllm_patch.py:245
↓ 2 callersFunctionremove_punc
(text)
eval/long_bench/metrics.py:21
↓ 2 callersMethodreset_stats
(self)
rag_baseline/rag_model.py:29
↓ 2 callersFunctionretrieve_relevant_sentences
( contexts, queries, token_budgets, )
rag_baseline/rag_utils.py:89
↓ 2 callersMethodspeculate
( self, execute_model_req: Optional[ExecuteModelRequest] = None )
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:91
↓ 2 callersFunctionwhite_space_fix
(text)
eval/long_bench/metrics.py:18
↓ 1 callersMethod_append_new_tokens
Given model output from a single run, append the tokens to the sequences. This is normally done outside of the worker, but it is requi
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:489
↓ 1 callersFunction_download_metric
()
eval/lm_eval_patch/scrolls/task.py:43
↓ 1 callersFunction_drop_duplicates_in_input
(untokenized_dataset)
eval/lm_eval_patch/scrolls/task.py:77
↓ 1 callersMethod_extract_prompt_execute_model_req
( self, execute_model_req: ExecuteModelRequest, look_ahead_cnt: int )
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:471
↓ 1 callersMethod_get_actual_look_ahead_cnts
( self, model_outputs: List[SamplerOutput], stop_token_ids: List[int] )
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:244
↓ 1 callersMethod_get_attention_scores
A caveat or potentially a good design here: For look ahead tokens, we also just attend to context, not g
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:375
↓ 1 callersMethod_get_kept_indices_from_token_importance
( self, token_importance: List[torch.Tensor] )
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:303
↓ 1 callersMethod_get_key_buffer
( self, slot_mapping: Optional[List[torch.Tensor]], kv_cache: List[torch.Tensor]
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:190
↓ 1 callersMethod_get_prefill_slot_mapping
( self, execute_model_req: Optional[ExecuteModelRequest] = None, )
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:445
↓ 1 callersMethod_get_prune_text
(self, sample)
eval/lm_eval_patch/scrolls/task.py:168
↓ 1 callersMethod_get_query_buffer
(self)
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:181
↓ 1 callersMethod_make_compute_metrics
(self, value)
eval/lm_eval_patch/scrolls/task.py:210
↓ 1 callersMethod_process_doc
(self, doc)
eval/lm_eval_patch/scrolls/task.py:391
↓ 1 callersMethod_process_doc
(self, doc)
eval/lm_eval_patch/scrolls/task.py:465
↓ 1 callersMethod_raise_if_unsupported
( self, execute_model_req: ExecuteModelRequest, )
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:520
↓ 1 callersMethod_reassemble_execute_model_req
( self, execute_model_req: ExecuteModelRequest, kept_indices: List[torch.LongTensor]
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:268
↓ 1 callersMethod_record_and_update_requests
( self, execute_model_req: ExecuteModelRequest )
speculative_prefill/vllm_patch/worker/spec_prefill_worker.py:183
↓ 1 callersMethod_token_importance_from_attn_scores
( self, attn_scores: List[torch.Tensor] )
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:332
↓ 1 callersFunctionadd_labels
(angles, values, labels, offset, ax)
eval/long_bench/calc_stats.py:58
↓ 1 callersMethoddoc_to_text
(self, doc)
eval/lm_eval_patch/scrolls/task.py:200
↓ 1 callersMethodfrom_path
(cls, config_path: Optional[str] = None)
speculative_prefill/vllm_patch/config.py:18
↓ 1 callersMethodfrom_seqs_and_pos_ids
Construct a :class:`SequenceData` instance from prompt and output token sequences.
speculative_prefill/vllm_patch/data/sequence.py:14
↓ 1 callersFunctionget_label_rotation
(angle, offset)
eval/long_bench/calc_stats.py:48
↓ 1 callersFunctionget_predictions
( model, sampling_params, data, prompt_format, dataset_name, output_path, no
eval/long_bench/pred_vllm.py:56
↓ 1 callersFunctionget_predictions
( model: RagLlama, max_gen, data, prompt_format, dataset_name, output_path,
eval/long_bench/pred_rag_llama.py:37
↓ 1 callersFunctioninit_spec_config
()
speculative_prefill/vllm_patch/config.py:52
↓ 1 callersFunctionllama_attn_forward_vllm
( vllm_version: str = "0.4.2", )
eval/long_bench/minference_vllm_patch.py:159
↓ 1 callersFunctionmain
(args: argparse.Namespace)
eval/minference_latency.py:21
↓ 1 callersFunctionminference_patch_vllm_executor
(config_file: str, patch_config={})
eval/long_bench/minference_vllm_patch.py:8
↓ 1 callersFunctionparse_args
()
eval/qps_client.py:149
↓ 1 callersFunctionparse_args
(args=None)
eval/long_bench/eval.py:36
↓ 1 callersFunctionparse_args
(args=None)
eval/long_bench/pred_vllm.py:18
↓ 1 callersFunctionparse_args
(args=None)
eval/long_bench/pred_rag_llama.py:16
↓ 1 callersFunctionpatch_data
()
speculative_prefill/vllm_patch/data/__init__.py:4
↓ 1 callersFunctionpatch_executor
()
speculative_prefill/vllm_patch/executor/__init__.py:5
↓ 1 callersFunctionpatch_scheduler
()
speculative_prefill/vllm_patch/scheduler.py:179
↓ 1 callersFunctionprepare_datasets
(category, num_samples)
eval/qps_client.py:66
↓ 1 callersMethodprint_stats
(self)
rag_baseline/rag_model.py:33
↓ 1 callersFunctionprocess_text
Processes the input text by adjusting punctuation spacing, normalizing spaces, and tokenizing it into sentences. Parameters: text (s
rag_baseline/rag_utils.py:21
↓ 1 callersMethodprune
Create a pruned version of a SCROLLS task dataset containing only inputs that are less than `max_tokens` when tokenized by each tokenizer
eval/lm_eval_patch/scrolls/task.py:171
↓ 1 callersFunctionremove_articles
(text)
eval/long_bench/metrics.py:15
↓ 1 callersFunctionretrieve_query_fn
(dataset_name)
rag_baseline/rag_utils.py:156
↓ 1 callersFunctionrouge_score
(prediction, ground_truth, **kwargs)
eval/long_bench/metrics.py:104
↓ 1 callersFunctionscorer
(dataset, predictions, answers, all_classes)
eval/long_bench/eval.py:63
↓ 1 callersFunctionscorer_e
(dataset, predictions, answers, lengths, all_classes)
eval/long_bench/eval.py:42
↓ 1 callersFunctionseed_everything
(seed)
eval/long_bench/pred_vllm.py:46
↓ 1 callersFunctionseed_everything
(seed)
eval/long_bench/pred_rag_llama.py:27
↓ 1 callersFunctionsend_query
( client, model, prompt, timeout, max_tokens, query_type )
eval/qps_client.py:21
↓ 1 callersFunctionsplit_num_cache_blocks_evenly
( base_model_cache_block_size_bytes: int, spec_model_cache_block_size_bytes: int, total_num_gpu_bl
speculative_prefill/vllm_patch/worker/spec_prefill_worker.py:21
↓ 1 callersFunctionsplit_sentences_batch
(texts)
rag_baseline/rag_utils.py:51
↓ 1 callersFunctionstream_results
()
eval/ruler_server.py:72
↓ 1 callersMethodupdate_stats
(self, ratio)
rag_baseline/rag_model.py:42
Method__init__
(self, config=None)
eval/lm_eval_patch/scrolls/task.py:118
Method__init__
( self, llama_model_name: str = 'meta-llama/Meta-Llama-3.1-8B-Instruct', percentage:
rag_baseline/rag_model.py:11
Method__init__
( self, base_model_worker: Worker, spec_model_worker: LookAheadSpecWorker, )
speculative_prefill/vllm_patch/worker/spec_prefill_worker.py:82
Method__post_init__
(self)
eval/lm_eval_patch/scrolls/task.py:228
Method__post_init__
(self)
speculative_prefill/vllm_patch/config.py:39
Method_compute_lens
Compute context length, sequence length and tokens for the given sequence data.
speculative_prefill/vllm_patch/data/input_builder.py:9
Method_driver_rank
(self)
speculative_prefill/vllm_patch/worker/spec_prefill_worker.py:216
Method_filter
(sample)
eval/lm_eval_patch/scrolls/task.py:182
Function_forward_with_query_dump
( self: LlamaAttention, positions: torch.Tensor, hidden_states: torch.Tensor, kv_cache: torch.
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:23
Method_get_keys_from_slot_mapping
( self, slot_mapping: List[torch.Tensor], layer_idx: int, kv_cache: List[to
speculative_prefill/vllm_patch/worker/look_ahead_spec_worker.py:425
Function_get_num_lookahead_slots
( self: scheduler.Scheduler, is_prefill: bool, enable_chunking: bool )
speculative_prefill/vllm_patch/scheduler.py:165
Method_get_prune_text
(self, doc)
eval/lm_eval_patch/scrolls/task.py:397
Method_get_worker_module_and_class
Patching this logic to consider using prefill speculator
speculative_prefill/vllm_patch/executor/gpu_executor.py:13
Method_process_doc
(self, doc)
eval/lm_eval_patch/scrolls/task.py:269
Method_process_doc
(self, doc)
eval/lm_eval_patch/scrolls/task.py:306
Method_process_doc
(self, doc)
eval/lm_eval_patch/scrolls/task.py:368
Method_process_doc
(self, doc)
eval/lm_eval_patch/scrolls/task.py:425
Function_schedule_prefills
Schedule sequence groups that are in prefill stage. Note that the current scheduler treats PREEMPTED_FOR_RECOMPUTE as a new prefill (
speculative_prefill/vllm_patch/scheduler.py:16
Method_scrolls_metrics
(self)
eval/lm_eval_patch/scrolls/task.py:231
next →1–100 of 156, ranked by callers