MCPcopy Create free account

hub / github.com/AtomaAI/atoma-infer / functions

Functions613 in github.com/AtomaAI/atoma-infer

⨍ Functions 613◇ Types & classes 173
↓ 1 callersMethodget_first_sequence
Retrieves a reference to the first sequence in the group, optionally filtered by status. # Arguments `status` - An optional `SequenceStatus` to filt
backends/vllm/src/sequence.rs:1512
↓ 1 callersMethodget_last_token_id
Returns the ID of the last generated token. # Returns - `Some(u32)`: The ID of the last generated token if any tokens have been generated. - `Some(u3
backends/vllm/src/sequence.rs:432
↓ 1 callersFunctionget_multiprocessor_count
(device_index: usize)
csrc/src/lib.rs:2201
↓ 1 callersMethodget_num_cpu_blocks
(&self)
backends/vllm/src/worker.rs:648
↓ 1 callersMethodget_num_new_tokens
Get the number of new tokens to be computed in the next iteration. This method determines how many tokens should be processed in the upcoming computa
backends/vllm/src/sequence.rs:931
↓ 1 callersMethodget_num_total_blocks
Gets total number of blocks
backends/vllm/src/block_allocator.rs:117
↓ 1 callersMethodget_output_text
Get `output_text`
backends/vllm/src/sequence.rs:547
↓ 1 callersMethodget_prefix_token_ids
Get prefix token IDs for the sequence. This function returns a tuple containing two vectors: 1. Prefix tokens from the prompt 2. Prefix tokens from t
backends/vllm/src/sequence.rs:309
↓ 1 callersMethodget_prompt_len
Returns the length of the prompt token ids. This method returns the number of tokens in the initial prompt. # Returns `usize` - The number of token
backends/vllm/src/sequence.rs:251
↓ 1 callersMethodget_seqs
Retrieves sequences from the `SequenceGroup` based on their status. # Arguments `status` - An optional `SequenceStatus` to filter the sequences. #
backends/vllm/src/sequence.rs:1422
↓ 1 callersFunctionhandle_generate_request
( app_state: &AppState, model: String, generate_request: GenerateRequest, )
server/src/server.rs:364
↓ 1 callersFunctionhandle_generate_stream_request
( app_state: &AppState, model: String, generate_request: GenerateRequest, )
server/src/server.rs:455
↓ 1 callersMethodhidden_dim
(&self)
backends/vllm/src/models/llama.rs:135
↓ 1 callersMethodhidden_dim
(&self)
backends/vllm/src/models/llama_nccl.rs:132
↓ 1 callersMethodis_full
Check if `token_ids` is full
backends/vllm/src/block.rs:76
↓ 1 callersMethodis_last_block_full
Checks if the last logical token block in the `Sequence` is full. # Returns - `true` if the last block exists and is full. - `false` if the last blo
backends/vllm/src/sequence.rs:686
↓ 1 callersMethodlogprobs
(&self)
server/src/api/chat_completions.rs:769
↓ 1 callersFunctionmake_tensor_with_pad
( x: Vec<Vec<D>>, max_length: usize, pad: D, device: &Device, )
backends/vllm/src/worker.rs:670
↓ 1 callersMethodmax_completion_tokens
(&self)
server/src/api/chat_completions.rs:777
↓ 1 callersMethodmessages
(&self)
server/src/api/chat_completions.rs:757
↓ 1 callersFunctionmessages_to_llama2_prompt
Function to convert a list of messages to a prompt string in Llama2 format.
server/src/api/chat_completions.rs:263
↓ 1 callersMethodnot_computed
Set `computed` to false
backends/vllm/src/block.rs:222
↓ 1 callersMethodnum_attention_heads
(&self)
backends/vllm/src/tests/mod.rs:50
↓ 1 callersMethodnum_hashed_tokens_of_block
Calculates the number of tokens that should be hashed for a given logical block index. This method is used to determine how many tokens should be con
backends/vllm/src/sequence.rs:594
↓ 1 callersMethodnum_key_value_heads
(&self)
models/src/llama.rs:55
↓ 1 callersFunctionnum_splits_heuristic
Find the number of splits that maximizes the occupancy. For example, if we have batch * n_heads = 48 and we have 108 SMs, having 2 splits (efficiency
csrc/src/lib.rs:2122
↓ 1 callersMethodnum_unfinished_sequences
Returns the number of unfinished sequences in the group. An unfinished sequence is one that has not yet reached a terminal state (e.g., not stopped,
backends/vllm/src/sequence.rs:1813
↓ 1 callersMethodpreempt_by_recompute
( &mut self, sequence_group: &mut SequenceGroup, )
backends/vllm/src/scheduler.rs:1860
↓ 1 callersMethodpreempt_by_swap
( &mut self, sequence_group: &mut SequenceGroup, blocks_to_swap_out: &mut HashMap<u32,
backends/vllm/src/scheduler.rs:1932
↓ 1 callersMethodprepare_input_tensors
( &self, sequence_groups_metadata: &[Arc<SequenceGroupMetadata>], )
backends/vllm/src/worker.rs:224
↓ 1 callersFunctionprepare_inputs
Prepares and tokenizes the input string, optionally truncating it. # Arguments `tokenizer` - The tokenizer to use for encoding the input. `input` -
backends/vllm/src/tokenizer.rs:185
↓ 1 callersMethodpresence_penalty
(&self)
server/src/api/chat_completions.rs:785
↓ 1 callersMethodprocess_generated_outputs
( &mut self, outputs: Vec<SequenceGroupOutput>, )
backends/vllm/src/llm_engine.rs:264
↓ 1 callersMethodprocess_received_request
( &self, request: GenerateRequest, )
backends/vllm/src/llm_service.rs:395
↓ 1 callersMethodprompt
Returns the prompt of the `SequenceGroup`. This function retrieves the prompt from the first sequence in the group. All sequences in a `SequenceGroup
backends/vllm/src/sequence.rs:1121
↓ 1 callersMethodread_lock
(&self)
backends/vllm/src/sequence.rs:1001
↓ 1 callersFunctionrelu_
csrc/kernels/utils.h:239
↓ 1 callersMethodremaining_budget_tokens
Computes the remaining number of budget tokens
backends/vllm/src/scheduler.rs:88
↓ 1 callersMethodremove_finished_sequences
(&mut self)
backends/vllm/src/scheduler.rs:2203
↓ 1 callersMethodreset
(&mut self)
backends/vllm/src/block_manager.rs:918
↓ 1 callersFunctionround_robin_task
Distributes tokenization requests among multiple workers using a round-robin algorithm. This function implements a simple round-robin scheduling stra
backends/vllm/src/tokenizer.rs:148
↓ 1 callersMethodrun
(mut self)
backends/vllm/src/llm_service.rs:267
↓ 1 callersMethodrun
(mut self)
backends/vllm/src/model_executor.rs:324
↓ 1 callersMethodrun
csrc/kernels/utils.h:113
↓ 1 callersFunctionrun_server
Runs the Axums server and manages its lifecycle, including graceful shutdown. This function sets up the HTTP router, starts the server, and handles t
server/src/server.rs:120
↓ 1 callersFunctionsample_outputs
()
backends/vllm/src/sequence.rs:2237
↓ 1 callersFunctionsampler_output
(sample_outputs: HashMap<u64, SequenceOutput>)
backends/vllm/src/sequence.rs:2253
↓ 1 callersMethodschedule_
(&mut self)
backends/vllm/src/scheduler.rs:1413
↓ 1 callersMethodschedule_chunked_prefill
(&mut self)
backends/vllm/src/scheduler.rs:1257
↓ 1 callersMethodschedule_default
(&mut self)
backends/vllm/src/scheduler.rs:1057
↓ 1 callersMethodseed
(&self)
server/src/api/chat_completions.rs:789
↓ 1 callersMethodset_computed
Set `computed`
backends/vllm/src/block.rs:187
↓ 1 callersMethodset_finished_time
Sets the finished time for the sequence group. This method updates the `finished_time` in the group's metrics to mark when the sequence group complet
backends/vllm/src/sequence.rs:1341
↓ 1 callersMethodset_last_accessed
Sets `last_accessed`
backends/vllm/src/block.rs:207
↓ 1 callersMethodsoftmax_scale
(&self)
backends/vllm/src/tests/mod.rs:66
↓ 1 callersFunctionstart_tokenizer_task
( tokenizer: Tokenizer, mut receiver: mpsc::UnboundedReceiver<EncodeTokenizerRequest>, span: Span,
backends/vllm/src/tokenizer.rs:95
↓ 1 callersMethodstop
(self)
backends/vllm/src/llm_service.rs:404
↓ 1 callersMethodstop
(&self)
server/src/api/chat_completions.rs:793
↓ 1 callersMethodstream
(&self)
server/src/api/chat_completions.rs:797
↓ 1 callersMethodswap_in
( &mut self, blocks_to_swap_in: &HashMap<u32, u32>, )
backends/vllm/src/worker.rs:602
↓ 1 callersMethodswap_in
( &mut self, seq_group: &mut SequenceGroup, )
backends/vllm/src/block_manager.rs:585
↓ 1 callersMethodswap_out
( &mut self, blocks_to_swap_out: &HashMap<u32, u32>, )
backends/vllm/src/worker.rs:619
↓ 1 callersMethodswap_out
( &mut self, seq_group: &mut SequenceGroup, )
backends/vllm/src/block_manager.rs:720
↓ 1 callersMethodtemperature
(&self)
server/src/api/chat_completions.rs:801
↓ 1 callersMethodto_generate_request
(self, request_id: String)
server/src/api/chat_completions.rs:891
↓ 1 callersMethodtokenize
( &self, input: String, truncate: Option<usize>, )
backends/vllm/src/validation.rs:69
↓ 1 callersMethodtop_p
(&self)
server/src/api/chat_completions.rs:805
↓ 1 callersMethodupdate_sequence
( &self, sequence: &Arc<RwLock<Sequence>>, sequence_output: &SequenceOutput, s
backends/vllm/src/llm_engine.rs:367
↓ 1 callersMethoduser
(&self)
server/src/api/chat_completions.rs:813
↓ 1 callersMethodvalidate_input
( &self, input: String, truncate: Option<usize>, max_new_tokens: Option<u32>,
backends/vllm/src/validation.rs:113
↓ 1 callersFunctionvalidate_with_schema
Given some `serde_json::Value` schema, construct a validator for some JSON instance returning a detailed error message upon failure. This can be used
server/src/api/validate_schema.rs:7
↓ 1 callersMethodverify_args
Verify arguments
backends/vllm/src/sampling_params.rs:254
↓ 1 callersMethodverify_beam_search
Verify beam search
backends/vllm/src/sampling_params.rs:347
↓ 1 callersMethodverify_greedy_sampling
Verify greedy sampling
backends/vllm/src/sampling_params.rs:391
↓ 1 callersMethodverify_non_beam_search
Verify non beam search
backends/vllm/src/sampling_params.rs:373
FunctionDEFINE_FLASH_FORWARD_KERNEL
csrc/kernels/flash_fwd_launch_template.h:38
MethodDropout
csrc/kernels/dropout.h:17
MethodMask
csrc/kernels/mask.h:117
Methodadd
(&mut self, block: PhysicalTokenBlock)
backends/vllm/src/evictor.rs:125
Methodadd
Adds a new `Sequence` to this `SequenceGroup`. If a sequence with the same ID already exists in the group, this method does nothing. # Arguments `s
backends/vllm/src/sequence.rs:1887
Methodalibi_slopes
(&self)
backends/vllm/src/models/llama.rs:125
Methodalibi_slopes
(&self)
backends/vllm/src/models/llama_nccl.rs:122
Methodapply_alibi
csrc/kernels/alibi.h:30
Methodapply_dropout
csrc/kernels/dropout.h:26
Functionapply_mask
csrc/kernels/mask.h:14
Methodapply_mask
csrc/kernels/mask.h:129
Functionapply_mask_causal
csrc/kernels/mask.h:75
Functionapply_mask_causal_w_idx
csrc/kernels/mask.h:84
Functionapply_mask_local
csrc/kernels/mask.h:38
Methodbfloat16_t>
csrc/kernels/utils.h:52
Methodblock_size
Getter for `block_size`
backends/vllm/src/config.rs:290
Methodblock_size
Getter for `block_size`
backends/vllm/src/sequence.rs:752
Methodcan_append_slots
Checks if new slots can be appended to the sequences in the given SequenceGroup. This method uses a heuristic to determine if there are enough free G
backends/vllm/src/block_manager.rs:254
Methodcannot_append_second_group
(sequence_group: &SequenceGroup, id: &str)
backends/vllm/src/scheduler.rs:3881
Functioncast_slice_mut
Cast a mutable slice of type T to a mutable slice of u8
csrc/src/ops.rs:224
FunctioncheckCudaErrors
Custom CUDA error checking function
csrc/kernels/flash_fwd_launch_template.h:25
Functioncombine_attn_seqk_parallel
csrc/kernels/flash_fwd_kernel.h:1132
Functioncompare_blocks
( tensor: &Tensor, src_block: usize, dst_block: usize, block_size: usize,
csrc/tests/cache_manager_tests.rs:230
Functioncompletion_handler
( app_state: State<AppState>, headers: HeaderMap, Json(request): Json<RequestBody>, )
server/src/server.rs:248
Functioncompute_attn
csrc/kernels/flash_fwd_kernel.h:1097
← previousnext →201–300 of 613, ranked by callers