MCPcopy Create free account

hub / github.com/AtomaAI/atoma-infer / functions

Functions613 in github.com/AtomaAI/atoma-infer

⨍ Functions 613◇ Types & classes 173
↓ 3 callersMethodinto_config
(self)
models/src/mistral.rs:37
↓ 3 callersMethodis_empty
Checks if the current instance is empty
backends/vllm/src/sequence.rs:2120
↓ 3 callersMethodmax_model_len
Getter for `max_model_len`
backends/vllm/src/config.rs:419
↓ 3 callersMethodmaybe_set_first_scheduled_time
Sets the first scheduled time and calculates the time spent in queue for request-level timings. This method updates the request metrics with the time
backends/vllm/src/sequence.rs:1316
↓ 3 callersMethodmock_schedule_running
( &mut self, running_queue: VecDeque<SequenceGroup>, budget: &mut Scheduli
backends/vllm/src/scheduler.rs:3844
↓ 3 callersMethodmock_schedule_swapped
( &mut self, budget: &mut SchedulingBudget, swapped_queue: VecDeque<Sequen
backends/vllm/src/scheduler.rs:4004
↓ 3 callersMethodref_count
Getter for `ref_count`
backends/vllm/src/block.rs:212
↓ 3 callersMethodreset_state_for_recompute
Resets the state for recomputation. This method should be called when a sequence needs to be restarted from the beginning, for example, when a sequen
backends/vllm/src/sequence.rs:421
↓ 3 callersFunctionrms_norm
(size: usize, eps: f64, vb: VarBuilder)
models/src/llama_nccl.rs:22
↓ 2 callersMethodabort_sequence_group
(&mut self, request_id: String)
backends/vllm/src/scheduler.rs:390
↓ 2 callersFunctionadd_new_token_to_output
(out: &SchedulerOutputs, token_id: u32)
backends/vllm/src/scheduler.rs:2319
↓ 2 callersMethodadd_token_id_to_seq
( &self, sequence_id: u64, token_id: u32, logprobs: HashMap<u32, LogProb>,
backends/vllm/src/sequence.rs:1157
↓ 2 callersMethodallocate_blocks
( &mut self, num_blocks: usize, device: &Device, #[cfg(feature = "nccl")] worl
backends/vllm/src/worker.rs:570
↓ 2 callersMethodappend_logical_block
Appends a new logical block to the sequence. This method creates a new `LogicalTokenBlock` and adds it to the `logical_token_blocks` vector. It's use
backends/vllm/src/sequence.rs:617
↓ 2 callersMethodappend_tokens_to_blocks
Appends tokens to the logical blocks of the `Sequence`. This method iterates through the given token IDs and appends them to the logical blocks. If n
backends/vllm/src/sequence.rs:648
↓ 2 callersMethodapply_rotary_embed
(&self, x: &Tensor, input_positions: &Tensor)
models/src/llama.rs:218
↓ 2 callersMethodapply_rotary_embed
(&self, x: &Tensor, input_positions: &Tensor)
models/src/phi3.rs:82
↓ 2 callersMethodapply_rotary_embed
(&self, x: &Tensor, input_positions: &Tensor)
models/src/llama_nccl.rs:46
↓ 2 callersMethodapply_rotary_embed
(&self, x: &Tensor, input_positions: &Tensor)
models/src/mistral.rs:82
↓ 2 callersMethodarrival_time
Retrieves the arrival time of the `SequenceGroup`. This method returns the time when the sequence group was initially received or created. # Returns
backends/vllm/src/sequence.rs:1358
↓ 2 callersMethodblock_size
Getter for `block_size`
backends/vllm/src/block.rs:66
↓ 2 callersFunctioncalculate_default_inv_freq
(cfg: &Config)
models/src/llama.rs:146
↓ 2 callersFunctioncalculate_swap_space
Calculate the swap space in bytes based on the available system memory and the specified fraction. This function determines the amount of memory to a
backends/vllm/src/config.rs:523
↓ 2 callersMethodcan_allocate
(&self, seq_group: &SequenceGroup)
backends/vllm/src/block_manager.rs:123
↓ 2 callersFunctioncast_slice
Cast a slice of type T to a slice of u8
csrc/src/ops.rs:231
↓ 2 callersFunctioncompute_total_block_memory_in_bytes
Computes the total memory required for a single block in the KV cache. This function calculates the total memory needed for both Key and Value tensor
backends/vllm/src/config.rs:708
↓ 2 callersMethodcomputed
Getter for `computed`
backends/vllm/src/block.rs:182
↓ 2 callersMethoddelay_factor
Getter for `delay_factor`
backends/vllm/src/config.rs:409
↓ 2 callersMethodforward
Flash attention forward pass # Arguments `q` - Query tensor with shape `[num_tokens, num_heads * head_size]` `k` - Key tensor with shape `[num_token
models/src/flash_attention.rs:322
↓ 2 callersMethodforward
( &mut self, x: &Tensor, input_positions: &Tensor, kv_cache: &Tensor,
models/src/llama_nccl.rs:81
↓ 2 callersMethodfree_block_table
(&mut self, block_table: &BlockTable)
backends/vllm/src/block_manager.rs:800
↓ 2 callersMethodfree_sequence
Frees blocks associated with a given sequence. This method releases the memory blocks allocated to a specific sequence, making them available for reu
backends/vllm/src/scheduler.rs:548
↓ 2 callersMethodfunction_call_string
(&self, model: Model)
server/src/api/chat_completions.rs:576
↓ 2 callersMethodget_last_token_id
Getter for last element in `token_ids`
backends/vllm/src/block.rs:128
↓ 2 callersMethodget_num_empty_slots
Get the number of additional token ids that can be added to the current `LogicalTokenBlock`
backends/vllm/src/block.rs:81
↓ 2 callersMethodget_num_gpu_blocks
(&self)
backends/vllm/src/worker.rs:652
↓ 2 callersMethodget_num_uncomputed_tokens
Computes the number of tokens that have not yet been processed. This method calculates the difference between the total length of the sequence (inclu
backends/vllm/src/sequence.rs:347
↓ 2 callersMethodget_output_len
Returns the length of the output token ids. This method returns the number of tokens that have been generated as output. # Returns `usize` - The nu
backends/vllm/src/sequence.rs:262
↓ 2 callersMethodget_physical_blocks
( &self, seq_group: &SequenceGroup, )
backends/vllm/src/block_manager.rs:465
↓ 2 callersMethodget_prompt_limit
Determines the maximum allowed length for prompts based on the scheduler configuration. # Returns - If chunked prefill is enabled: Returns `max_mode
backends/vllm/src/scheduler.rs:2120
↓ 2 callersMethodget_unfinished_sequences
Retrieves all unfinished sequences from the `SequenceGroup`. This method filters and returns a vector of all sequences that have not yet finished pro
backends/vllm/src/sequence.rs:1589
↓ 2 callersMethodhidden_dim
(&self)
backends/vllm/src/tests/mod.rs:46
↓ 2 callersMethodlast_accessed
Getter for `last_accessed`
backends/vllm/src/block.rs:202
↓ 2 callersMethodmock_append_slots
( &mut self, sequence_group: &SequenceGroup, blocks_to_copy: &mut HashMap<
backends/vllm/src/scheduler.rs:3981
↓ 2 callersMethodmock_schedule_prefill
( &mut self, mut waiting_queue: VecDeque<SequenceGroup>, budget: &mut Sche
backends/vllm/src/scheduler.rs:3719
↓ 2 callersMethodmodel
(&self)
server/src/api/chat_completions.rs:753
↓ 2 callersMethodnext_token_chooser_params
Returns the next token chooser parameters for this `SequenceGroup`. # Returns A clone of the `NextTokenChooserParameters` associated with this group
backends/vllm/src/sequence.rs:1928
↓ 2 callersMethodnum_batched_tokens
Number of batched tokens
backends/vllm/src/scheduler.rs:141
↓ 2 callersMethodnum_cpu_blocks
Getter for `num_cpu_blocks`
backends/vllm/src/config.rs:320
↓ 2 callersMethodnum_current_sequences
Number of current sequences
backends/vllm/src/scheduler.rs:146
↓ 2 callersMethodnum_gpu_blocks
Getter for `num_gpu_blocks`
backends/vllm/src/config.rs:315
↓ 2 callersMethodnum_hashed_tokens
Getter for `num_hashed_tokens`
backends/vllm/src/block.rs:197
↓ 2 callersMethodnum_hidden_layers
(&self)
backends/vllm/src/tests/mod.rs:54
↓ 2 callersMethodnum_kv_heads
(&self)
backends/vllm/src/tests/mod.rs:58
↓ 2 callersMethodpassed_delay
(&mut self, now: Instant)
backends/vllm/src/scheduler.rs:2077
↓ 2 callersMethodprompt_token_ids
Returns a clone of the prompt token IDs. # Returns A `Vec<u32>` containing the token IDs of the prompt.
backends/vllm/src/sequence.rs:443
↓ 2 callersFunctionquad_allreduce_
csrc/kernels/softmax.h:38
↓ 2 callersFunctionreshape_flatten_thread_tile
csrc/kernels/utils.h:332
↓ 2 callersFunctionscale_apply_exp2
csrc/kernels/softmax.h:66
↓ 2 callersMethodschedule
( &mut self, )
backends/vllm/src/scheduler.rs:1455
↓ 2 callersMethodschedule_running
( &mut self, running_queue: VecDeque<SequenceGroup>, budget: &mut SchedulingBudget,
backends/vllm/src/scheduler.rs:612
↓ 2 callersMethodset_ref_count_by
Sets the `ref_count` by `value`
backends/vllm/src/block.rs:232
↓ 2 callersFunctionsilu
(xs: &Tensor)
models/src/llama_nccl.rs:27
↓ 2 callersMethodstep
(&mut self)
backends/vllm/src/llm_engine.rs:216
↓ 2 callersMethodstopping_params
Returns the stopping criteria parameters for this `SequenceGroup`. This method provides access to the stopping criteria used to determine when to hal
backends/vllm/src/sequence.rs:1940
↓ 2 callersMethodswap_blocks
Initiates a swap blocks operation on the current CUDA device
models/src/flash_attention.rs:281
↓ 2 callersMethodswap_in
( &mut self, sequence_group: &mut SequenceGroup, blocks_to_swap_in: &mut HashMap<u32,
backends/vllm/src/scheduler.rs:2032
↓ 2 callersMethodvalidate
( &self, request: GenerateRequest, )
backends/vllm/src/validation.rs:187
↓ 2 callersMethodverify_cache_dtype
Verify `CacheConfig` cache dtype
backends/vllm/src/config.rs:280
↓ 2 callersMethodwrite_lock
(&self)
backends/vllm/src/sequence.rs:1011
↓ 1 callersMethodabort_sequence_groups
Aborts multiple sequence groups at once. This method iterates through the provided request IDs and aborts each corresponding sequence group. It's a c
backends/vllm/src/scheduler.rs:473
↓ 1 callersMethodaccess_all_blocks_in_sequence
Updates the last access time for all blocks associated with a given sequence. This method iterates through all blocks in the sequence's block table a
backends/vllm/src/block_manager.rs:1006
↓ 1 callersMethodalibi_slopes
(&self)
backends/vllm/src/tests/mod.rs:38
↓ 1 callersMethodappend_tokens
(&mut self, token_ids: &[u32])
backends/vllm/src/block.rs:105
↓ 1 callersMethodcache_swap
( &mut self, blocks_to_swap_in: &HashMap<u32, u32>, blocks_to_swap_out: &HashMap<u32,
backends/vllm/src/worker.rs:195
↓ 1 callersFunctioncalculate_num_cpu_blocks
Calculates the number of CPU blocks that can be allocated in the swap space. This function determines how many blocks of the KV cache can be stored i
backends/vllm/src/config.rs:667
↓ 1 callersFunctioncalculate_num_gpu_blocks
Calculates the number of GPU blocks that can be used for the KV cache based on available GPU memory. This function determines the number of blocks th
backends/vllm/src/config.rs:590
↓ 1 callersMethodcan_append_slots
(&self, sequence_group: &SequenceGroup)
backends/vllm/src/scheduler.rs:1647
↓ 1 callersMethodcan_swap_in
( &self, seq_group: &SequenceGroup, )
backends/vllm/src/block_manager.rs:522
↓ 1 callersMethodcan_swap_out
(&self, seq_group: &SequenceGroup)
backends/vllm/src/block_manager.rs:667
↓ 1 callersFunctioncheck_schema
Used in tandem with a schema file, this will check if there are changes to the JSON API schema, and show a diff if so. If there are changes, running t
server/src/api/chat_completions.rs:1155
↓ 1 callersFunctioncompile_cuda_files
(build_dir: &Path)
csrc/build.rs:117
↓ 1 callersFunctioncompute_selected_token_indices
Computes selected token indices, for each sequence in the batch. For a given sequence, the associated selected token index should correspond to the ri
backends/vllm/src/worker.rs:688
↓ 1 callersFunctionconvert_layout_acc_dropout
csrc/kernels/utils.h:216
↓ 1 callersMethodcopy_blocks
(&mut self, blocks_to_copy: Tensor)
backends/vllm/src/worker.rs:636
↓ 1 callersMethoddecrease_ref_count
Decreases the reference count by 1. # Returns - `Ok(())` if the reference count was successfully decreased. - `Err(BlockError::ReferenceCountError)`
backends/vllm/src/block.rs:247
↓ 1 callersMethodeos_token_ids
(&self)
backends/vllm/src/tests/mod.rs:42
↓ 1 callersMethoderr_string
(&self)
server/src/api/validate_schema.rs:47
↓ 1 callersMethodexecute_model
( &mut self, request: ExecuteModelRequest, )
backends/vllm/src/worker.rs:111
↓ 1 callersFunctionfa_acausal
(q: &Tensor, k: &Tensor, v: &Tensor, softmax_scale: f32)
csrc/tests/flash_attn_tests.rs:19
↓ 1 callersMethodfinished_reason
Returns the reason why the sequence finished, if applicable. # Returns - `Some(String)`: A string describing the reason for finishing, if the sequenc
backends/vllm/src/sequence.rs:118
↓ 1 callersFunctionflash_attn
Flash-attention v2 layer. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Multi-query and grouped-query attenti
csrc/src/lib.rs:392
↓ 1 callersFunctionflash_attn_varlen
Flash-attention v2 layer with variable-length batching. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Multi-q
csrc/src/lib.rs:1160
↓ 1 callersFunctionflash_attn_varlen
()
csrc/tests/flash_attn_tests.rs:97
↓ 1 callersFunctionflash_attn_varlen_with_block_table
Flash-attention v2 layer with variable-length batching. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Multi-q
csrc/src/lib.rs:1392
↓ 1 callersFunctionflash_attn_varlen_with_block_table
()
csrc/tests/flash_attn_tests.rs:142
↓ 1 callersMethodforward
( &mut self, _: &Tensor, _: &Tensor, _: &Tensor, _: Vec<&mut Tensor>,
backends/vllm/src/tests/mod.rs:140
↓ 1 callersMethodfree
(&mut self, block: SyncPhysicalTokenBlock)
backends/vllm/src/block_allocator.rs:89
↓ 1 callersMethodfrequency_penalty
(&self)
server/src/api/chat_completions.rs:761
↓ 1 callersMethodget_finished_sequences
Retrieves all finished sequences from the `SequenceGroup`. This method filters and returns a vector of all sequences that have completed processing.
backends/vllm/src/sequence.rs:1625
← previousnext →101–200 of 613, ranked by callers