Check if a tensor is a Transformer Engine Float8Tensor

helper function for linear cross entropy.

This is an analogue of torch.narrow for ShardedTensors. Narrowing assumes that we narrow a local tensor on each rank. This has conseq

Setup model and optimizer.

Swap x[src_idxs] and x[dst_idxs]

This method verifies a result and possibly triggers a re-run. Args: result: result to verify. rejection_func: functio

Wait until communication (and associated computation) is completed.

Returns a new ShardedBase instance with data=None.

Write the index (.idx) file Args: sequence_lengths (List[int]): The length of each sequence sequence_modes (Optional

Determine the type, nargs, and choices settings for this argument. Args: config_type: attribute type from dataclass

Use multi-tensor-applier to copy values from one list to another. We don't have a bfloat16 implementation so for now if the overflow_buf

Add additional arguments for using TensorRT Model Optimizer (modelopt) features.

Reduce a tensor of losses across all GPUs.

Backward function.

Condition TE init_method on config.perform_initialization.

Create embeddings with marker values at specific positions. Args: num_embeddings: Number of embeddings to create mark

Create model, DDP wrapper, and optimizer. Args: model_class: Model class to instantiate clip_grad: Optional gradient

Disable the feature flag.

Specify the tensors to be released after offloading. forced_released_tensors is a list of tensors to be released after offloading. The te

Mark the start of a layer group and prepare for offload/reload.

Create a WeightedMultiTask from a config list. Args: config: List of dicts with keys: - agent_type: String path t

Utility to generate all the output tokens and probabilities for the prompts. This utility generates the output tokens for a static batch. It

Get default save sharded strategy.

Retrieve a tensor item managed by the `DataParallelBuffer` instance, i.e. get all the item data stored in this sharded or unsharded b

Get the local model for a certain (pp,ep,tp).

Compute per-token log-probabilities (and optionally entropy) on responses. For each sample, extracts response-aligned logits and tokens, then com

Returns the underlying fault tolerance client instance Returns: RankMonitorClient: rank monitor client instance, or None if FT was not in

Extract all text content from the app's UI.

Get TRTLLM Config and Converted Model Weights This function returns the trtllm model weights as a list. There are two modes for conve

Which module to use for layer norm

Take a string math problem and return the prompt. Supports requesting tagged or boxed answers. Supports chat mode prompts.

Finalizes active async save calls. Args: blocking (bool, optional): if True, will wait until all active requests are done. Ot

Set the module to save the original input tensors. Some transformer-engine modules would save the quantized tensors by default in fp8 traini

Block forward pre-hook for certain configurations.

Set lr for all parameters groups. Args: increment (int): number of steps to increment

Converts list of tokens to it's ids.

Auxiliary function for checking the values in the checkpoint and setting them. Args: cls_value (float): class value

Communicate tensors between stages. Used as helper method in other communication methods that are used in megatron/schedules.py. Args

Make sure the input variable is not None.

Convert dataclass name into appropriate argparse flag name. Args: config_attr_name: dataclass attribute name prefix:

Method to drop all the buffered microbatches and jump to the next iteration.

Gather tensors across all ranks in the context parallel group. The first dimension of the output tensor will be the `response_length`.

Ensure `metadata` is a dict containing `dp_cp_group` entry. If `metadata` is None, a new dict is returned with `dp_cp_group` set. If `metadat

Args: path (str): path to tokenizer file with metadata.json in folder. metadata_path (Optional[str]): path to the tok

Return the aux loss coeff for the given auxiliary loss type. If the auxiliary loss type is not found, return 0.0.

Get synchronized time across ranks.

Simple linear layer with weight initialization.

Get the global RL runtime state.

Function to get the rotary sequence length. Args: inference_context : Used during Inference time transformer (Transfo

Get pytorch version from __version__; if not available use pip's. Use caching.

Initialize Megatron, tokenizers, and datasets.

Create the RNG tracker. 'use_te_rng_tracker' determines whether to use Megatron or TransformerEngine's implementation. In particular, Transfor

Load a sharded (untar .nemo or megatron --use-dist-ckpt) or unsharded checkpoint. Essentially, the function is detecting whether the checkpoint i

Merges dicts and lists recursively.

Shift start/end indexes to start at new start index. Both start and end indexes will be shifted by [new start] - [old start]. Args:

Open a file with the appropriate method based on whether MSC is enabled.

If distributed is initialized, print only on last rank.

Receive a message over the multiprocessing queue.

Wrapper for autograd function: forward: RS, backward AG <fisrt dim>

run() is the protocol between our schedule logic and model, which is used to schedule the forward and backward schedule plans for the models.

Samples the logits to generate outputs Given the logits of the last token, this function samples it according to the parameters defin

Unset a key in the global config file.

Initialize affine weight for model parallel. Build the master weight on all processes and scatter the relevant chunk.

Make a POST request to the specified endpoint with the given payload. Args: endpoint: The API endpoint to call payloa