Linear layer with row parallelism. The linear layer is defined as Y = XA + b. A is parallelized along its first dimension and X along its sec

The attention layer implementation.

Embedding parallelized in the vocabulary dimension. Adapted from torch.nn.Embedding, note that we pad the vocabulary size to make sure it is

Parallelized LM head. Output logits weight matrices used in the Sampler. The weight and bias tensors are padded to make sure they are divisib

Linear layers for the attention's QKV transformation. Linear layers for the linear transformation of the query, key, and value vectors in the

Replicated linear layer. Args: input_size: input dimension of the linear layer. output_size: output dimension of the linear layer

Result of streaming incremental parsing.

Packed linear layers with column parallelism. Similar to ColumnParallelLinear, but the weight matrix is concatenated along the output dimensi

Linear layer with column parallelism. The linear layer is defined as Y = XA + b. A is parallelized along its second dimension as A = [A_1, ..

A class that manages prompt templates and keeps all conversation history.

A layer that pools specific information from hidden states. This layer does the following: 1. Extracts specific tokens or aggregates data base

A single multimodal data, from a single image/video/audio or others

Note: The communicator now only run up to 1 in-flight request at any time.

Parameter class for linear layer weights. Uses both column and row parallelism.

Parameter class for scales where the number of scales is equivalent to the number of logical matrices in fused linear layers (e.g. for QK

The input and output status of a request.

Linear method without quantization.

Result of streaming incremental parsing.

Store all inputs of a forward pass.

The sampling parameters. See docs/backend/sampling_params.md or https://docs.sglang.ai/backend/sampling_params.html for the document

Simple encapsulation of the parsed ToolCall result for easier usage in streaming contexts.

r""" Multi-headed attention without any cache, mostly used for ViT. Args: use_qkv_parallel (bool, optional): If True, use QKV-pa

FlashAttention backend implementation. Note about the init: - If no spec decoding - FlashAttentionBackend will be init once when the

Metadata to be init once in the model forward pass, each layer's forward pass can reuse the metadata. For each init metadata function, we wil

In streaming scenarios, each time new_text is received, it calls multi_format_parser.parse_streaming_increment and returns the resulting norm

Lazy import to make `import sglang` run faster.

Result of evaluating a single sample

Parameter class for weight scales loaded for weights with channel-wise quantization. Equivalent to _ColumnvLLMParameter.

A class that manages completion prompt templates. only for code completion currently.

The entry point to the inference engine. - The engine consists of three components: 1. TokenizerManager: Tokenizes the requests and

FusedMoE layer for MoE models. This layer contains both MergedColumnParallel weights (gate_up_proj / w13) and RowParallelLinear weights (down

In this pattern, data tokens should be represented as repetitions of a single token e.g. <image><image>....<image>, or <audio><audio>...<audio>

In this pattern, data tokens should be enclosed by special token pairs (e.g. <image>...</image>, data_token_pairs) This strategy should be applie

Parser that handles both streaming and non-streaming scenarios for extracting reasoning content from model outputs. Args: model_

Store all information of a batch on the scheduler.

RotaryEmbedding extended with YaRN method. Credits to Peng et al. github.com/jquesnelle/yarn

PyTorch ProcessGroup wrapper for a group of processes. PyTorch ProcessGroup is bound to one specific communication backend, e.g. NCCL

download_dir: Directory to download and load the weights, default to the default cache directory of huggingface. load_format: The for