Root mean square normalization. Computes x -> w * x / sqrt(E[x^2] + eps) where w is the learned weight. Refer to https://arxiv.org/abs/1910.0

Linear layer with row parallelism. The linear layer is defined as Y = XA + b. A is parallelized along its first dimension and X along its sec

Sampling parameters for text generation. Overall, we follow the sampling parameters from the OpenAI text completion API (https://platform.ope

Replicated linear layer. Args: input_size: input dimension of the linear layer. output_size: output dimension of the linear layer

Attention layer. This class takes query, key, and value tensors as input. The input tensors can either contain prompt tokens or generation tok

1. Discard the hidden states that are not used for sampling (i.e., all tokens except the final one in each prompt). 2. Compute the lo

Linear layers for the attention's QKV transformation. Linear layers for the linear transformation of the query, key, and value vectors in the

Samples the next tokens from the model's outputs. This layer does the following: 3. Apply presence, frequency and repetition penalties. 4

Embedding parallelized in the vocabulary dimension. Adapted from torch.nn.Embedding, note that we pad the vocabulary size to make sure it is

Linear method without quantization. Args: separate_bias_add: If true, add bias separately after matrix multipl

A worker class that executes (a partition of) the model on a GPU. Each worker is associated with a single GPU. The worker is responsible for

Manages free physical token blocks for a device. The allocator maintains a list of free blocks and allocates a block when requested. When a b

Metadata for input sequences. Used in PagedAttention. Args: prompt_lens: Lengths of prompts. slot_mapping: The address to write t

An LLM for generating texts from given prompts and sampling parameters. This class includes a tokenizer, a language model (possibly distributed

Packed linear layers with column parallelism. Similar to ColumnParallelLinear, but the weight matrix is concatenated along the output dimensi

Parallelized LM head. Output logits weight matrices used in the Sampler. The weight and bias tensors are padded to make sure they are divisib

Data associated with a sequence. Args: prompt_token_ids: The token IDs of the prompt. Attributes: prompt_token_ids: The tok

Metadata for a sequence group. Used to create `InputMetadata`. Args: request_id: The ID of the request. is_prompt: Whether the r

An activation function for SwiGLU. The function computes x -> silu(x[:d]) * x[d:] where d = x.shape[-1] // 2. Shapes: x: (batch_size

Linear method for AWQ. Args: quant_config: The AWQ quantization config.

A stream of RequestOutputs for a request that can be iterated over asynchronously.

Manages the mapping between logical and physical token blocks.

Configuration for the KV cache. Args: block_size: Size of a cache block in number of tokens. gpu_memory_utilization: Fraction of

Manages the KV cache. This class is responsible for initializing and managing the GPU and CPU KV caches. It also provides methods for perform

Linear layer with column parallelism. The linear layer is defined as Y = XA + b. A is parallelized along its second dimension as A = [A_1, ..

The output data of one completion output of a request. Args: index: The index of the output in the request. text: The generated o

RotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla

Arguments for vLLM engine.

If the input tensors contain prompt tokens, the layout is as follows: |<--------------- num_prompt_tokens -------------->| |<--prompt_0-

Linear method for GPTQ. Args: quant_config: The GPTQ quantization config.

RotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendev

A block that stores a contiguous chunk of tokens from left to right. Logical blocks are used to represent the states of the corresponding phy

Configuration for the model. Args: model: Name or path of the huggingface model to use. tokenizer: Name or path of the huggingfac

Adds logging prefix to newlines to align multi-line messages.

Configuration for the distributed execution. Args: pipeline_parallel_size: Number of pipeline parallel groups. tensor_parallel_si

Represents the state of a block in the KV cache.

Synchronous abstraction for tracking requests.

Original rotary positional embedding.

Metadata for input sequences. Used in sampler. Args: seq_groups: List of (seq_ids, sampling_params). seq_data: Seq_id -> Sequence

An activation function with post-scale parameters. This is used for some quantization methods like AWQ.

Scheduler configuration. Args: max_num_batched_tokens: Maximum number of tokens to be processed in a single iteration.

Stores the data, status, and block information of a sequence. Args: seq_id: The ID of the sequence. prompt: The prompt of the seq

A group of sequences that are generated from the same prompt. Args: request_id: The ID of the request. seqs: The list of sequence

The model output associated with a sequence group.

The model output associated with a sequence. Args: parent_seq_id: The ID of the parent sequence (for forking in beam search).

Linear method for SqueezeLLM. Args: quant_config: The SqueezeLLM quantization config.

If the input tensors contain prompt tokens, the layout is as follows: |<--------------- num_prompt_tokens --------------->| |<--prompt_0

RotaryEmbedding extended with YaRN method. Credits to Peng et al. github.com/jquesnelle/yarn

Config class for AWQ. Reference: https://arxiv.org/abs/2306.00978

Result for BlockSpaceManager.can_allocate 1. Ok: seq_group can be allocated now. 2. Later: seq_group cannot be allocated. The capacity

Arguments for asynchronous vLLM engine.

An asynchronous wrapper for LLMEngine. This class is used to wrap the LLMEngine class to make it asynchronous. It uses asyncio to create a ba

Construct a Baichuan tokenizer. Based on byte-level Byte-Pair-Encoding. Args: vocab_file (`str`): Path to the vocabulary