This is a Module Specification dataclass. Specification defines the location of the module (to import dynamically) or the imported module its

Base checkpointing related exception

Definition of a SubPolicy. A SubPolicy consists of two augmentation operations, each of those parametrized as operation, probability, magnitu

Transformer class.

fused operation: scaling + mask + softmax Args: input_in_fp16: flag to indicate if input in fp16 data format. input_in_bf16:

Represents a mapping between a local object and a global object. Global object is assumed to consist of many local objects distributed betwee

Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).

The low-level interface dataset class Args: path_prefix (str): The index (.idx) and data (.bin) prefix multimodal (bool, optiona

Linear Embedding

Image to Patch Embedding

Transformer class.

A range represents a start and end points for indexing a shard from a full tensor.

Rotary Embedding for language model. Args: kv_channels (int): Projection weights dimension in multi-head attention. Obtained from transfo

Vision Transformer Model.

Forward step function with all the communications. We use a class here to hide the inference parameters from the outside caller.

Dataset to convert GPT tokens to text. Args: gpt_dataset (MultiSplitGPTDataset): GPT dataset, which outputs GPT token samples. gp

Apply Gaussian Blur to the PIL image.

A generic data loader where the images are arranged in this way: :: root/dog/xxx.png root/dog/xxy.png root/dog/[...]/xxz.png

Language model embeddings. Args: config (TransformerConfig): config object with all necessary configs for TransformerBlock vocab_

Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MoeDecoderLayer`] Args: config: MoeConfig

Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MoleDecoderLayer`] Args: config: MoleConfig

BERT-based encoder for queries or contexts used for learned information retrieval.

Documents backends used in the checkpoint. Checkpoint config keeps track of formats used for storing the sharded tensors (sharded_backend) a

An efficient implementation of the Experts layer using CUTLASS GroupedGEMM. This class is designed to execute multiple experts in parallel, t

Dataset containing sentences and their blocks for an inverse cloze task.

BERT-based encoder for queries or blocks used for learned information retrieval.

Definition of an ImageNetPolicy. Implements a fixed AutoAugment data augmentation policy targeted at ImageNet training by randomly applying a

Object that should not be stored in a checkpoint, but restored locally. Wrapping any object inside the state dict with LocalNonpersitentObject

Perform forward pass separately on each resolution input. The inputs corresponding to a single resolution are clubbed and single forward

Serializable data structure for holding data for blocks -- embeddings and necessary metadata for Retriever

Parallel self-attention layer abstract class. Self-attention layer takes input with size [s, b, h] and returns output of the same size.

MLP. MLP will take the input with h hidden state, project it to 4*h hidden dimension, perform nonlinear transformation, and project the s

Represents a mapping between a local tensor and a global tensor. Global tensor is assumed to consist of many local tensors distributed betwee

Allows to apply transformations to tensors before/after serialization. The essence of those transformations is that they can be applied to o

A conditional wrapper to initialize an instance of Transformer-Engine's `LayerNorm` or `RMSNorm` based on input

Original BERT wordpiece tokenizer.

Object class to read the index (.idx) file Args: idx_path (str): The path to the index file multimodal (bool): Whether the datas

A basic Swin Transformer layer for one stage. Args: dim (int): Number of input channels. input_resolution (tuple[int]): Input resu

A basic Swin Transformer layer for one stage. Args: dim (int): Number of input channels. input_resolution (tuple[int]): Input re

Runs basic tokenization (punctuation splitting, lower casing, etc.).

Masked LM head for Bert Args: config: TransformerConfig object mpu_vocab_size: model parallel size of vocabulary. paralle

Masked LM head for Bert. Args: hidden_size: hidden size config (TransformerConfig): TransformerConfig object

Bert-based module for Biencoder model.

Builder class for the BlendedDataset and MegatronDataset classes Args: cls (Type[MegatronDataset]): The class to instantiate, must inheri

A struct for fully describing a fixed-size block of data as used in REALM :param start_idx: for first sentence of the block :param end_idx: f

A struct for fully describing a fixed-size block of data as used in REALM :param start_idx: for first sentence of the block :param end_idx: f

Bucket to keep track of a subset of the model's gradients. Provides functionality to register when params in the bucket have grads ready to b

CLIP ViT vision model. Args: transformer_config (TransformerConfig): Transformer config transformer_layer_spec (ModuleSpec): Spec

ChainedOptimizer is designed for a collection of optimizers. These optimizers are responsible for different parts of multiple models for

Constant grad scaler (loss scale is never adjusted regardless of NaNs seen in gradients).

Cross-attention layer class Cross-attention layer takes input with size [s, b, h] and context with size [s, b, h] and returns output of the s

Tracker for the cuda RNG states. Using the `add` method, a cuda rng state is initialized based on the input `seed` and is assigned to `name`.

Custom handler to filter out logging from code outside of Megatron Core, and dump to stdout.

Dataset for iterating chunks. Args: db_path (str): Path of HDF5-format chunk database. indexed_datasets (List[IndexedDataset]

Grad scaler with dynamic scale that gets adjusted during training. Reduces loss scale by `backoff_factor` if `hysteresis` number of NaNs are

Language model embeddings. Args: hidden_size: hidden size vocab_size: vocabulary size max_sequence_length: maximum size o

Float32 optimizer. Args: optimizer (torch.optim.Optimizer): base optimizer such as Adam or SGD. config (OptimizerConfig): configu

Async-enabled implementation of FileSystemWriter using file IO. This class doesn't spawn the async process itself, relies on the external as

Implement the scaled dot product attention with softmax. Arguments --------- softmax_scale: The temperature to use for the softmax att

Float16 optimizer for fp16 and bf16 data types. Args: optimizer (torch.optim.Optimizer): base optimizer such as Adam or SGD. conf