MCPcopy Create free account

hub / github.com/MLSysU/TD-Pipe / functions

Functions572 in github.com/MLSysU/TD-Pipe

⨍ Functions 572◇ Types & classes 128↳ Endpoints 6
↓ 22 callersMethodget_seqs
( self, status: Optional[SequenceStatus] = None, )
TD_Pipe/sequence.py:285
↓ 18 callersFunctionget_tensor_model_parallel_world_size
Return world size for the tensor model parallel group.
TD_Pipe/model_executor/parallel_utils/parallel_state.py:125
↓ 18 callersFunctionset_weight_attrs
Set attributes on a weight tensor. This method is used to set attributes on a weight tensor. This method will not overwrite existing attribut
TD_Pipe/model_executor/utils.py:17
↓ 13 callersFunctioncreate_error_response
(status_code: HTTPStatus, message: str)
TD_Pipe/entrypoints/openai/api_server.py:100
↓ 13 callersFunctioninit_logger
(name: str)
TD_Pipe/logger.py:50
↓ 13 callersMethodis_finished
(self)
TD_Pipe/sequence.py:212
↓ 11 callersFunctionget_tensor_model_parallel_rank
Return my rank for the tensor model parallel group.
TD_Pipe/model_executor/parallel_utils/parallel_state.py:140
↓ 11 callersMethodremove
(self, seq_id: int)
TD_Pipe/sequence.py:323
↓ 9 callersFunctionis_hip
()
TD_Pipe/utils.py:33
↓ 8 callersFunctiondivide
Ensure that numerator is divisible by the denominator and return the division value.
TD_Pipe/model_executor/parallel_utils/utils.py:16
↓ 8 callersFunctionrandom_uuid
()
TD_Pipe/utils.py:51
↓ 7 callersMethod_run_workers
Runs the given method on all workers.
TD_Pipe/engine/llm_engine.py:864
↓ 7 callersMethodfree
(self, block: PhysicalTokenBlock)
TD_Pipe/core/block_manager.py:46
↓ 7 callersMethodget_beam_search_score
Calculate the beam search score with length penalty. Adapted from https://github.com/huggingface/transformers/blob/ccb92be23def445f2
TD_Pipe/sequence.py:193
↓ 7 callersMethodget_from_keys
Get a value from the model's quantization config.
TD_Pipe/model_executor/layers/quantization/base_config.py:45
↓ 7 callersMethodget_len
(self)
TD_Pipe/sequence.py:172
↓ 6 callersMethodallocate
(self)
TD_Pipe/core/block_manager.py:39
↓ 6 callersMethodgenerate
Generates the completions for the input prompts. NOTE: This class automatically batches the given prompts, considering the memory con
TD_Pipe/entrypoints/llm.py:112
↓ 6 callersMethodget_num_free_blocks
(self)
TD_Pipe/core/block_manager.py:53
↓ 5 callersMethod__init__
( self, config: OPTConfig, parallel_config: ParallelConfig, attention_backend:
TD_Pipe/model_executor/models/opt.py:287
↓ 5 callersMethod__init__
( self, num_experts: int, hidden_size: int, intermediate_size: int, li
TD_Pipe/model_executor/models/mixtral.py:62
↓ 5 callersMethodadd
(self, seq: Sequence)
TD_Pipe/sequence.py:318
↓ 5 callersMethodcopy
(self, src_to_dsts: Dict[int, List[int]])
TD_Pipe/worker/cache_engine.py:150
↓ 5 callersMethodfree_seq
(self, seq: Sequence)
TD_Pipe/core/scheduler.py:449
↓ 4 callersMethod__init__
( self, input_size: int, output_size: int, bias: bool = True, skip_bia
TD_Pipe/model_executor/layers/linear.py:87
↓ 4 callersMethod__init__
( self, hidden_size: int, intermediate_size: int, hidden_act: str, lin
TD_Pipe/model_executor/models/qwen2.py:59
↓ 4 callersMethod__init__
( self, hidden_size: int, intermediate_size: int, hidden_act: str, lin
TD_Pipe/model_executor/models/llama.py:57
↓ 4 callersFunction_is_cuda
()
setup.py:21
↓ 4 callersMethod_process_model_outputs
( self, output: SamplerOutput, scheduler_outputs: SchedulerOutputs)
TD_Pipe/engine/llm_engine.py:606
↓ 4 callersMethodconvert_tokens_to_string
Converts a sequence of tokens (string) in a single string.
TD_Pipe/transformers_utils/tokenizers/baichuan.py:127
↓ 4 callersMethodget_last_token_id
(self)
TD_Pipe/sequence.py:184
↓ 4 callersFunctionget_pipeline_model_parallel_world_size
Return world size for the pipeline model parallel group.
TD_Pipe/model_executor/parallel_utils/parallel_state.py:130
↓ 4 callersFunctionget_tensor_model_parallel_group
Get the tensor model parallel group the caller rank belongs to.
TD_Pipe/model_executor/parallel_utils/parallel_state.py:111
↓ 4 callersFunctionhf_model_weights_iterator
( model_name_or_path: str, cache_dir: Optional[str] = None, load_format: str = "auto", revisio
TD_Pipe/model_executor/weight_utils.py:191
↓ 4 callersMethodnum_seqs
(self, status: Optional[SequenceStatus] = None)
TD_Pipe/sequence.py:304
↓ 4 callersMethodunfreeze_seq_groups
( self, scheduled_seq_groups: Iterable[SequenceGroup])
TD_Pipe/core/scheduler.py:175
↓ 4 callersMethodupdate_kv_cache
(self, input_len, predicted_len)
TD_Pipe/core/scheduler.py:125
↓ 3 callersMethod__init__
( self, head_size: int, rotary_dim: int, max_position_embeddings: int,
TD_Pipe/model_executor/layers/rotary_embedding.py:49
↓ 3 callersMethod_compute_inv_freq
Compute the inverse frequency.
TD_Pipe/model_executor/layers/rotary_embedding.py:68
↓ 3 callersMethod_set_config_defaults
( self, config: Dict[str, Any], config_defaults: Dict[str, Any])
TD_Pipe/transformers_utils/configs/mpt.py:157
↓ 3 callersMethodabort
Abort a request. Abort a submitted request. If the request is finished or not found, this method will be a no-op. Args:
TD_Pipe/engine/async_llm_engine.py:448
↓ 3 callersMethodabort_request
Abort a request during next background loop iteration.
TD_Pipe/engine/async_llm_engine.py:133
↓ 3 callersMethodadd_cli_args
Shared CLI arguments for vLLM engine.
TD_Pipe/engine/arg_utils.py:43
↓ 3 callersMethodapply_weights
Apply the weights to the input tensor.
TD_Pipe/model_executor/layers/linear.py:32
↓ 3 callersFunctioncheck_length
( request: Union[ChatCompletionRequest, CompletionRequest], prompt: Optional[str] = None, prompt_i
TD_Pipe/entrypoints/openai/api_server.py:141
↓ 3 callersMethodcreate_weights
Create weights for a linear layer.
TD_Pipe/model_executor/layers/linear.py:24
↓ 3 callersMethodfrom_engine_args
Creates an LLM engine from the engine arguments.
TD_Pipe/engine/llm_engine.py:313
↓ 3 callersMethodget_cumulative_logprob
(self)
TD_Pipe/sequence.py:190
↓ 3 callersMethodget_head_size
(self)
TD_Pipe/config.py:199
↓ 3 callersFunctionget_ip
()
TD_Pipe/utils.py:60
↓ 3 callersFunctionget_lock
(model_name_or_path: str, cache_dir: Optional[str] = None)
TD_Pipe/model_executor/weight_utils.py:29
↓ 3 callersMethodget_num_kv_heads
Returns the number of KV heads per GPU.
TD_Pipe/config.py:237
↓ 3 callersMethodget_num_layers
(self, parallel_config: "ParallelConfig")
TD_Pipe/config.py:252
↓ 3 callersMethodget_output_len
(self)
TD_Pipe/sequence.py:178
↓ 3 callersFunctionget_pipeline_model_parallel_rank
Return my rank for the pipeline model parallel group.
TD_Pipe/model_executor/parallel_utils/parallel_state.py:145
↓ 3 callersMethodget_prompt_len
(self)
TD_Pipe/sequence.py:175
↓ 3 callersFunctionget_rope
( head_size: int, rotary_dim: int, max_position: int, base: int, is_neox_style: bool = Tru
TD_Pipe/model_executor/layers/rotary_embedding.py:332
↓ 3 callersFunctionin_wsl
()
TD_Pipe/utils.py:55
↓ 3 callersMethodput
(self, item: RequestOutput)
TD_Pipe/engine/async_llm_engine.py:51
↓ 3 callersFunctiontensor_model_parallel_all_reduce
All-reduce the input tensor across model parallel group. NOTE: This operation is applied in-place on the input tensor.
TD_Pipe/model_executor/parallel_utils/communication_op.py:69
↓ 2 callersMethod_abort
Abort a request. Abort a submitted request. If the request is finished or not found, this method will be a no-op. Args:
TD_Pipe/engine/async_llm_engine.py:466
↓ 2 callersMethod_add_request
( self, prompt: Optional[str], sampling_params: SamplingParams, prompt_token_i
TD_Pipe/entrypoints/llm.py:161
↓ 2 callersMethod_allocate_kv_cache
Allocates KV cache on the specified device.
TD_Pipe/worker/cache_engine.py:94
↓ 2 callersMethod_append_logical_block
(self)
TD_Pipe/sequence.py:139
↓ 2 callersMethod_append_tokens_to_blocks
(self, token_ids: List[int])
TD_Pipe/sequence.py:146
↓ 2 callersFunction_async_h2d
(data: list, dtype, pin_memory)
TD_Pipe/worker/model_runner.py:450
↓ 2 callersFunction_convert_tokens_to_string_with_added_encoders
( tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast], output_tokens: List[str], skip_s
TD_Pipe/transformers_utils/tokenizer.py:70
↓ 2 callersMethod_free_block_table
(self, block_table: BlockTable)
TD_Pipe/core/block_manager.py:265
↓ 2 callersFunction_get_bin_counts_and_mask
( tokens: torch.Tensor, vocab_size: int, num_seqs: int, )
TD_Pipe/model_executor/layers/sampler.py:88
↓ 2 callersMethod_get_physical_blocks
( self, seq_group: SequenceGroup)
TD_Pipe/core/block_manager.py:190
↓ 2 callersMethod_get_prefill_time
(self, seq_len)
TD_Pipe/core/scheduler.py:269
↓ 2 callersMethod_preempt_by_recompute
( self, seq_group: SequenceGroup, )
TD_Pipe/core/scheduler.py:506
↓ 2 callersMethod_run_engine
(self, use_tqdm: bool)
TD_Pipe/entrypoints/llm.py:173
↓ 2 callersMethod_swap
( self, src: List[KVCache], dst: List[KVCache], src_to_dst: Dict[int, int],
TD_Pipe/worker/cache_engine.py:126
↓ 2 callersFunction_yarn_find_correction_dim
(num_rotations: int, dim: int, base: float = 10000
TD_Pipe/model_executor/layers/rotary_embedding.py:227
↓ 2 callersMethodadd_request
Add a request to be sent to the engine on the next background loop iteration.
TD_Pipe/engine/async_llm_engine.py:116
↓ 2 callersMethodappend_token_id
( self, token_id: int, logprobs: Dict[int, float], )
TD_Pipe/sequence.py:162
↓ 2 callersFunctioncheck_model
(request)
TD_Pipe/entrypoints/openai/api_server.py:131
↓ 2 callersMethodclear_kv_cache
(self)
TD_Pipe/core/scheduler.py:131
↓ 2 callersFunctioncompletion_stream_generator
()
TD_Pipe/entrypoints/openai/api_server.py:290
↓ 2 callersMethodcreate_engine_configs
( self, )
TD_Pipe/engine/arg_utils.py:206
↓ 2 callersFunctioncreate_logprobs
Create OpenAI-style logprobs.
TD_Pipe/entrypoints/openai/api_server.py:185
↓ 2 callersFunctioncreate_stream_response_json
( index: int, text: str, logprobs: Optional[LogProbs] = None, finish_reason: O
TD_Pipe/entrypoints/openai/api_server.py:543
↓ 2 callersMethodfinish
(self)
TD_Pipe/engine/async_llm_engine.py:56
↓ 2 callersMethodfork
(self, parent_seq: Sequence, child_seq: Sequence)
TD_Pipe/core/block_manager.py:182
↓ 2 callersMethodfork_seq
(self, parent_seq: Sequence, child_seq: Sequence)
TD_Pipe/core/scheduler.py:446
↓ 2 callersMethodfrom_cli_args
(cls, args: argparse.Namespace)
TD_Pipe/engine/arg_utils.py:199
↓ 2 callersMethodfrom_config
(cls, config: Dict[str, Any])
TD_Pipe/model_executor/layers/quantization/awq.py:57
↓ 2 callersMethodget_finished_seqs
(self)
TD_Pipe/sequence.py:301
↓ 2 callersMethodget_min_capability
(self)
TD_Pipe/model_executor/layers/quantization/awq.py:45
↓ 2 callersMethodget_num_empty_slots
(self)
TD_Pipe/block.py:30
↓ 2 callersMethodget_num_free_gpu_blocks
(self)
TD_Pipe/core/block_manager.py:289
↓ 2 callersFunctionget_open_port
()
TD_Pipe/utils.py:64
↓ 2 callersFunctionget_role
()
TD_Pipe/entrypoints/openai/api_server.py:284
↓ 2 callersMethodget_sliding_window
(self)
TD_Pipe/config.py:190
↓ 2 callersMethodget_supported_head_sizes
()
TD_Pipe/model_executor/layers/attention/ops/paged_attn.py:15
↓ 2 callersMethodget_token_ids
(self)
TD_Pipe/sequence.py:181
↓ 2 callersFunctionget_tokenizer
Gets a tokenizer for the given model name via Huggingface.
TD_Pipe/transformers_utils/tokenizer.py:13
↓ 2 callersFunctioninitialize_cluster
Initialize the distributed cluster probably with Ray. Args: parallel_config: The configurations for parallel execution. engine_us
TD_Pipe/engine/ray_utils.py:54
↓ 2 callersMethodsample
( self, logits: torch.Tensor, sampling_metadata: SamplingMetadata, )
TD_Pipe/model_executor/models/opt.py:351
next →1–100 of 572, ranked by callers