MCPcopy Create free account

hub / github.com/AtomaAI/atoma-infer / functions

Functions613 in github.com/AtomaAI/atoma-infer

⨍ Functions 613◇ Types & classes 173
Functioncompute_attn_1rowblock
csrc/kernels/flash_fwd_kernel.h:57
Functioncompute_attn_1rowblock_splitkv
csrc/kernels/flash_fwd_kernel.h:505
Functioncompute_attn_splitkv
csrc/kernels/flash_fwd_kernel.h:1118
Methodcompute_full_blocks_in_sequence
Marks full blocks in a `Sequence` as computed. This function iterates through the blocks of a given sequence and marks them as computed if they are f
backends/vllm/src/block_manager.rs:1045
Methodconfig
(&self)
backends/vllm/src/models/llama.rs:119
Methodconfig
(&self)
backends/vllm/src/models/llama_nccl.rs:116
Methodconfig_7b_v1
()
models/src/llama.rs:102
Methodconfig_7b_v2
()
models/src/llama.rs:120
Methodcontrol
()
server/src/api/chat_completions.rs:820
Functionconvert_layout_acc_Aregs
csrc/kernels/utils.h:198
Functionconvert_type
csrc/kernels/utils.h:227
Functionconvert_type_relu
csrc/kernels/utils.h:255
Methodcopy_blocks
Initiates a copy blocks operation on the current CUDA device
models/src/flash_attention.rs:294
Functioncopy_blocks_t
Launches the `copy_blocks_kernel` on the given `key_caches` and `value_caches`, following the `block_mapping`, to copy the blocks on both `key_cache`
csrc/src/cache_manager.rs:166
Functioncopy_rotary_contiguous
csrc/kernels/rotary.h:85
Functioncopy_rotary_interleaved
csrc/kernels/rotary.h:22
Functioncopy_w_min_idx
csrc/kernels/utils.h:413
Functioncp_async_wait
csrc/kernels/utils.h:285
Methodcpu_fwd
(&self, _s: &CpuStorage, _l: &Layout)
models/src/multi_gpu.rs:71
Methodcpu_fwd
( &self, _: &mut candle_core::CpuStorage, _: &candle_core::Layout, _: &candle_
csrc/src/ops.rs:30
Methodcpu_fwd
( &self, _: &CpuStorage, _: &Layout, _: &CpuStorage, _: &Layout,
csrc/src/lib.rs:350
Methodcreate_empty
Create an empty `Self` instance
backends/vllm/src/scheduler.rs:173
Methodcuda_fwd
( &self, s: &candle_core::CudaStorage, l: &Layout, )
models/src/multi_gpu.rs:75
Methodcuda_fwd
( &self, dst_c: &mut CudaStorage, dst_l: &Layout, src_c: &CudaStorage,
csrc/src/ops.rs:40
Methodcuda_fwd
( &self, q: &candle_core::CudaStorage, q_l: &Layout, k: &candle_core::CudaStor
csrc/src/lib.rs:362
Methodcuda_fwd_t
( &self, q: &candle_core::CudaStorage, q_l: &Layout, k: &candle_core::CudaStor
csrc/src/lib.rs:31
Methoddecoded_token
Getter for `decoded_token`
backends/vllm/src/sequence.rs:52
Methoddefault
()
backends/vllm/src/sampling_params.rs:31
Methoddefault
()
backends/vllm/src/evictor.rs:78
Functiondefault_parameters
()
backends/vllm/src/types.rs:75
Functiondefault_rope
()
models/src/llama.rs:60
Methoddeserialize
(deserializer: D)
server/src/api/chat_completions.rs:476
Functiondeserialize_assistant_message
()
server/src/api/chat_completions.rs:1276
Functiondeserialize_message_content_array
()
server/src/api/chat_completions.rs:1350
Functiondeserialize_message_content_text
()
server/src/api/chat_completions.rs:1339
Functiondeserialize_request_body_basic
()
server/src/api/chat_completions.rs:1193
Functiondeserialize_system_message
()
server/src/api/chat_completions.rs:1212
Functiondeserialize_tool_message
()
server/src/api/chat_completions.rs:1318
Functiondeserialize_user_message
()
server/src/api/chat_completions.rs:1232
Methodempty
Creates an empty instance of `Self`
backends/vllm/src/sequence.rs:2113
Methodeos_token_ids
(&self)
backends/vllm/src/models/llama.rs:128
Methodeos_token_ids
(&self)
backends/vllm/src/models/llama_nccl.rs:125
Methodeq
(&self, other: &Self)
backends/vllm/src/sequence.rs:182
Methodevict
(&mut self)
backends/vllm/src/evictor.rs:88
Methodfetch
( api_key: String, cache_dir: T, model_id: String, revision: String, )
backends/vllm/src/tests/mod.rs:78
Methodfetch
( api_key: String, cache_dir: T, model_id: String, revision: String, )
backends/vllm/src/models/llama.rs:39
Methodfetch
( api_key: String, cache_dir: T, model_id: String, revision: String, )
backends/vllm/src/models/llama_nccl.rs:39
Functionflash_attn_acausal
()
csrc/tests/flash_attn_tests.rs:33
Functionflash_attn_alibi
Flash-attention v2 layer. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Multi-query and grouped-query attenti
csrc/src/lib.rs:464
Functionflash_attn_alibi_windowed
Flash-attention v2 layer. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Multi-query and grouped-query attenti
csrc/src/lib.rs:506
Functionflash_attn_alibi_windowed_with_softcap
( q: &Tensor, k: &Tensor, v: &Tensor, alibi_slopes: &Tensor, softmax_scale: f32, windo
csrc/src/lib.rs:552
Functionflash_attn_kv_cache
Flash-attention v2 layer with key and value tensors cached. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Mul
csrc/src/lib.rs:1907
Functionflash_attn_kv_cache
()
csrc/tests/flash_attn_tests.rs:196
Functionflash_attn_kv_cache_alibi
Flash-attention v2 layer with key and value tensors cached. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Mul
csrc/src/lib.rs:1989
Functionflash_attn_kv_cache_alibi_windowed
Flash-attention v2 layer with key and value tensors cached. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Mul
csrc/src/lib.rs:2036
Functionflash_attn_kv_cache_windowed
Flash-attention v2 layer with key and value tensors cached. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Mul
csrc/src/lib.rs:1949
Functionflash_attn_varlen_alibi
Flash-attention v2 layer with variable-length batching. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Multi-q
csrc/src/lib.rs:1268
Functionflash_attn_varlen_alibi_windowed
Flash-attention v2 layer with variable-length batching. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Multi-q
csrc/src/lib.rs:1328
Functionflash_attn_varlen_full
Flash-attention v2 layer with variable-length batching. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Multi-q
csrc/src/lib.rs:1464
Functionflash_attn_varlen_windowed
Flash-attention v2 layer with variable-length batching. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Multi-q
csrc/src/lib.rs:1218
Functionflash_attn_windowed
Flash-attention v2 layer. This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`. Multi-query and grouped-query attenti
csrc/src/lib.rs:432
Methodfmt
(&self, f: &mut std::fmt::Formatter<'_>)
backends/vllm/src/sequence.rs:1946
Methodfmt
(&self, f: &mut std::fmt::Formatter<'_>)
server/src/api/chat_completions.rs:132
Methodfmt
(&self, f: &mut fmt::Formatter<'_>)
models/src/phi3.rs:128
Methodfmt
(&self, f: &mut fmt::Formatter<'_>)
models/src/mistral.rs:130
Methodfork
(&self, new_sequence_id: u64)
backends/vllm/src/sequence.rs:903
Methodforward
( &mut self, input: &Tensor, input_positions: &Tensor, selected_token_position
backends/vllm/src/models/llama.rs:102
Methodforward
( &mut self, input: &Tensor, input_positions: &Tensor, selected_token_position
backends/vllm/src/models/llama_nccl.rs:99
Methodforward
(&self, x: &Tensor)
models/src/multi_gpu.rs:16
Methodfree_sequences
( &mut self, request_id: String, sequences_ids: &[u64], sequence_status: Seque
backends/vllm/src/scheduler.rs:506
Methodfrom
(value: ExecuteModelRequest)
backends/vllm/src/tests/mod.rs:126
Methodfrom
(value: ValidationError)
server/src/api/validate_schema.rs:32
Methodfrom_env_file
Creates a new instance of `ModelConfig` from a `.env` file.
backends/vllm/src/config.rs:86
Methodfrom_file_path
Creates a new instance of `ModelsConfig` from a toml file.
backends/vllm/src/config.rs:73
Methodfrom_file_path
Creates a new instance of self, from a file path
backends/vllm/src/model_executor.rs:118
Methodfrom_file_path
(_: &PathBuf)
backends/vllm/src/tests/mod.rs:70
Methodfrom_sequence_group
Creates a new `Self` instance from a `SequenceGroup`
backends/vllm/src/llm_engine.rs:543
Methodget_beam_search_score
Calculate the beam search score with length penalty. Adapted from https://github.com/huggingface/transformers/blob/ccb92be23def445f2afdea94c31286f84
backends/vllm/src/sequence.rs:862
Methodget_config
(&self)
models/src/llama.rs:501
Methodget_kv_cache_shape
Returns the KV cache shape for the given model configurations.
models/src/flash_attention.rs:239
Methodget_last_latency
Calculates the latency since the last token generation and updates the last token time. This method performs two main tasks: 1. It calculates the dur
backends/vllm/src/sequence.rs:1237
Functionget_lse_tile
csrc/kernels/flash_fwd_kernel.h:36
Methodget_num_free_blocks
Gets number of free blocks
backends/vllm/src/block_allocator.rs:112
Methodget_priority
(now: Instant, sequence_group: &SequenceGroup)
backends/vllm/src/policy.rs:58
Methodget_token_ids
Returns all token ids, including both prompt and output. This method combines the prompt token ids and the output token ids into a single vector. #
backends/vllm/src/sequence.rs:274
Methodgets_all_computed_blocks
( &self, sequence: Sequence, )
backends/vllm/src/block_manager.rs:1095
Methodgpu_memory_utilization
Getter for `gpu_memory_utilization`
backends/vllm/src/config.rs:295
Methodhalf_t>
csrc/kernels/utils.h:34
Methodhandle_outputs
( &mut self, outputs: Result<Vec<SequenceGroupOutput>, EngineError>, )
backends/vllm/src/llm_engine.rs:157
Methodhandle_request
( &mut self, request: GenerateRequest, )
backends/vllm/src/llm_service.rs:318
Methodhas_unfinished_sequences
Checks if there are any unfinished sequences in the scheduler. This method returns true if any of the scheduler's queues (waiting, running, or swappe
backends/vllm/src/scheduler.rs:574
Methodhash_of_block
Computes the hash of a block given its logical index. This function calculates a hash based on the prefix tokens up to the given logical block index.
backends/vllm/src/sequence.rs:571
Functionhealthz
()
server/src/server.rs:202
Methodinto_response
(self)
server/src/server.rs:179
Methodload
( _: Self::C, _: &Device, _: DType, _: &ModelFilePaths, )
backends/vllm/src/tests/mod.rs:104
Methodload
( config: Self::C, device: &Device, dtype: DType, file_paths: &ModelFilePaths,
backends/vllm/src/models/llama.rs:73
Methodload
( config: Self::C, device: &Device, dtype: DType, file_paths: &ModelFilePaths,
backends/vllm/src/models/llama_nccl.rs:73
Methodload
(vb: VarBuilder, cfg: &Config, dtype: DType, device: &Device)
models/src/llama.rs:316
Methodload
(vb: VarBuilder, cfg: &Phi3Config, _dtype: DType, _device: &Device)
models/src/phi3.rs:373
Methodload
(vb: VarBuilder, comm: Rc<Comm>)
models/src/multi_gpu.rs:19
← previousnext →301–400 of 613, ranked by callers