Configuration object for megatron-core transformers. The initialization function has an argument for each parameter, including those in Model

This is a Module Specification dataclass. Specification defines the location of the module (to import dynamically) or the imported module its

TemporaryDirectory with a fully named directory. Empties the dir if not empty.

Inference parameters sent along with the prompts. This class contains request-level attributes that control the sampling techniques used when

GPT-2 Language model.

Base checkpointing related exception

r"""N-dimensional communication grid. Manages an arbitrary number of parallelisms as a hyperrectangle. Each dimension is given a name at init

Unified process group collection for transformer model parallelism, gradient communication, and finalization. Fields use init=False and must

Utility that adds an argument group to an ArgumentParser based on the attributes of a dataclass. This utility uses dataclass metadata including t

The dataclass for ModuleSpecs of MLP submodules including linear fc1, activation function, linear fc2.

Configuration class for specifying the submodules of a transformer layer. This class defines the structure and default implementations for v

Configuration for DistributedDataParallel.

Builder class for the BlendedDataset and MegatronDataset classes Args: cls (Type[MegatronDataset]): The class to instantiate, must inheri

Configuration class for specifying the submodules of a self-attention.

Definition of a SubPolicy. A SubPolicy consists of two augmentation operations, each of those parametrized as operation, probability, magnitu

Transformer class.

This is an internal class, not for use outside this module Attributes: _rank (int): rank for the value _value (float) : the value

Base optimizer configuration object.

DDP wrapper which stores grads in contiguous buffers. Also has option of overlapping communication with backprop computation by breaking up f

Class for one inference request Containing relevant data for an dynamic inference request

Layer context that can be matched to a quantization config.

Represents a mapping between a local object and a global object. Global object is assumed to consist of many local objects distributed betwee

Configuration object for megatron-core Multi-Latent Attention (MLA) transformers. The initialization function has an argument for each parameter,

Mixture of Experts layer. This layer implements a Mixture of Experts model, where each token is routed to a subset of experts. This implement

The sample generated

MLA Self-attention layer class Self-attention layer takes input with size [s, b, h] and returns output of the same size.

Base configuration for Megatron Core The initialization function has an argument for each parameter.

Config for the model inference wrapper NOTE : All the arguments here are obtained from arguments.py file

The global state for the generation process.

Image transformation.

The low-level interface dataset class Args: path_prefix (str): The index (.idx) and data (.bin) prefix multimodal (bool): Whethe

parameters to TEDotProductAttention and fused rope kernels for the `thd` (packed) sequence format

Dataclass for specifying the submodules of a transformer block. This class defines the structure for configuring the layers and normalizatio

fused operation: scaling + mask + softmax Args: input_in_fp16: flag to indicate if input in fp16 data format. input_in_bf16:

Self-attention layer class Self-attention layer takes input with size [s, b, h] and returns output of the same size.

Wraps arbitrary load strategy and distributes the load during `load`. See `load` method docs for details. Args: strategy (LoadSharde

Inference wrapper for GPT model. The wrapper prepares the model for inference, provides the required input data, and runs the forward pass A

Class for one inference request Containing relevant data for an inference request

A docstring section.

Tensor Parallel Muon optimizer.

Tokenized representation of a language-based Rollout.

Configuration object for Megatron Core datasets

A simple counter class This class is responsible for assigning request ids to incoming requests

Minimal test app for providing Textual context.

Base configuration for Megatron Core Export These parameters control the export setting for trtllm

Float 16 Module. Attributes: config (TransformerConfig): Transformer config fp16 (bool) : Specifies if the model runs in fp16 mod

Anneals learning rate and weight decay Args: optimizer (MegatronOptimizer): the optimizer to be used init_lr (float): initial lea

Config options for different prompt formats.

Rotary Embedding for language model. Args: kv_channels (int): Projection weights dimension in multi-head attention. Obtained

Arg meta.

Object that should not be stored in a checkpoint, but restored locally. Wrapping any object inside the state dict with LocalNonpersistentObject

Test configuration args.

Inference context that is passed to the main model in order to efficiently calculate and store the KV cache during inference. The dynamic inf

The dynamic inference engine. This engine allows requests of varying length to be dynamically added and removed in each inference step. In co

Configuration object for Megatron Core GPT datasets

GPT-specific pipeline customizations. Args: src (Union[ModelMeta, Tuple]): Model meta for loading. dst (Union[ModelMeta, Tuple]):

The text generation controller (the main sampling loop) This class tokenizes the input, runs inference, samples from logits, and detokenizes the

Pipeline Communicator between two modules with different(TP/DP/PP/CP). BridgeCommunicator: - Initialize the communicator between a pair of so

Batch dimensions for dynamic inference. Attributes: token_count : number of total input tokens prefill_req_count : number of pref

A protocol for providing the submodules used in Spec building.

A single transformer layer. Transformer layer takes input with size [s, b, h] and returns an output of the same size.

Tracker for the cuda RNG states. Using the `add` method, a cuda rng state is initialized based on the input `seed` and is assigned to `name`.

Wraps arbitrary strategy and distributes the save during `save`. The save distribution happens without any *data* communication. Only the *me

Global model statistics tracker with optional limits.

LLaVA multi-modal model. Args: language_transformer_config (TransformerConfig): Transformer config for the language model. langua

A protocol for providing Local submodules used in Spec building.

Mamba language model. Args: config (TransformerConfig): Model config mamba_stack_spec (ModuleSpec): Specifies the modules to use

Communicator for a multi-module pipeline.

Raised for conditions that can be handled by the agent.

A wrapper class for data iterators that adds replay capability. Args: iterable: data iterator that needs the replay capability. m

Allows to apply transformations to tensors before/after serialization. The essence of those transformations is that they can be applied to op

Static inference context that is passed to the main model in order to efficiently manage the KV cache during inference. Args: max_bat

The Megatron core backend constructor This is the backend that does a simple forward pass on the model. Supports any model that is callable (

Yarn Rotary Embedding for language model. Args: kv_channels (int): Projection weights dimension in multi-head attention. Obtained from

Bert Language model.

A container for holding data in Fully Sharded Data Parallel (FSDP) training. Attributes: data (torch.Tensor): A tensor containing th

Configuration class for specifying the submodules of a cross-attention.

A class that manages the data parallel buffer for Fully Sharded Data Parallel (FSDP) training. It has two operating modes given a bucket of m

Distributed optimizer, for all data types (fp16, bf16, and fp32). See __init__() below for argument details.

A dummy state class that holds various attention-related parameters. This class is used to simulate the state object that would normally be

Fully Sharded Data Parallel (FSDP) wrapper for the Megatron model.

Multimodal In/Out Model supporting arbitrary combinations of modalities. .. warning:: **EXPERIMENTAL**: This class is experimental, still

Configuration for a multi-modal model. Args: language_model_spec (ModuleSpec): Specification for the language model m