Set global variables, initialize distributed, and set autoresume and random seeds. `allow_no_cuda` should not be set unless using megatron for

Train index on a representative set of vectors. Args: config (RetroPreprocessingConfig): Retro preprocessing config.

args: y_data : 4-d tensor in [batch_size, chs, img_rows, img_cols] dim : int return 3-d [int] y_data

Allows to construct a *flattened* ShardedTensor given offset specified in process ranks. Args: key (str): data (torch

Write the index (.idx) file Args: sequence_lengths (List[int]): The length of each sequence sequence_modes (Optional

Set random seed for reproducability.

Compile C++ helper functions at runtime. Make sure this is invoked on a single process.

Wrapper for autograd function: forward: AG, backward: RS <first dim>

Get an attribute from a wrapped model. If return_model_obj is true, return the object that has the 'attr' attribute; otherwise, return the att

Return caller's rank for the pipeline-model-parallel group.

Returns (potentially) a sub-tensor from the self.buffer for the given shape.

Wraps tensors from transformer layers with ShardedTensor or ShardedObject. For a given `state_dict`, wraps: - all _extra_states with ShardedO

args: y_true : [int] 3-d in [batch_size, img_rows, img_cols] y_pred : [int] 3-d in [batch_size, img_rows, img_cols] return [f

Unique id of the sharded tensor data. Should yield the same value for same data replicated on different ranks. Args: sharded_tensor

Reroute to the appropriate apply_rotary_pos_emb function depending on fused/unfused kernels, or bshd (conventional) / thd (packed seq) format

Condition TE init_method on config.perform_initialization.

Forward step for passed-in model. If it is the first stage, the input tensor is obtained from the data_iterator. Otherwise, the passed-in inp

Get default save sharded strategy.

Helper function to return the singleton instance of the rerun machine.

Set args, tokenizer, tensorboard-writer, adlr-autoresume, and timers.

Function to share the input embeddings and output logit weights.

Set lr for all parameters groups. Args: increment (int): number of steps to increment

Update number of microbatches. Args: consumed_samples (int): Number of samples consumed. consistency_check (bool, opt

Auxiliary function for checking the values in the checkpoint and setting them. Args: cls_value (float): class value

Communicate tensors between stages. Used as helper method in other communication methods that are used in megatron/schedules.py. Args:

Returns the active time.

Broadcast tensor values from last stage into the first stage.

Simple linear layer with weight initialization.

Return world size for the pipeline-model-parallel group.

Get rng states. Copy the dictionary so we have direct pointers to the states, not just a pointer to the dictionary.

Return tensorboard writer. It can be None so no need to check if it is initialized.

Return tensorboard writer. It can be None so no need to check if it is initialized.

Import a named object from a module in the context of this function. TODO: make this importer module more robust, at least make sure there ar

Check if a tensor is a Transformer Engine Float8Tensor

Finalizes active async save calls. Args: blocking (bool, optional): if True, will wait until all active requests are done. O

This is an analogue of torch.narrow for ShardedTensors. Narrowing assumes that we narrow a local tensor on each rank. This has conseq

Save the auxiliary loss for logging. Args: name (str): The name of the loss. loss (torch.Tensor): The loss tensor. layer_n

See megatron.legacy.model.transformer.set_input_tensor()

Returns a new ShardedBase instance with data=None.

Make sure the input variable is not None.

args: y_true : 3-d ndarray in [batch_size, img_rows, img_cols] y_pred : 3-d ndarray in [batch_size, img_rows, img_cols] retur

Retrieve keys from the model that end with a specified suffix.

Initialize affine weight for model parallel. Build the master weight on all processes and scatter the relevant chunk.

Reduce a tensor of losses across all GPUs.

Data loader. Note that batch-size is the local (per GPU) batch-size.

Build dataloader given an input dataset.

Initialize tokenizer.

Remove new lines and multiple spaces and adjust end of sentence dot.

Convert model weights to trtllm model weights This method goes through each layer in the model state dict and converts to equivalent trtllm m

Multiple with alpha and add beat with clip.

Convert list of token IDs to plaintext.

Tracker file rescords the latest chckpoint during training to restart from.

Return the global GlobalMemoryBuffer object

Returns the config attribute, allowed to return None

Get TRTLLM Config and Converted Model Weights This function returns the trtllm model weights as a list. There are two modes for conve

Note that this call will sync across all ranks.

Note that this call will sync across all ranks.

A patched version of put, implementing PR 3544 and PR 4295.

Wrapper for autograd function: forward: RS, backward AG <fisrt dim>

Helper function to rename model layer names to TRTLLM Layer names We go through each layer (keys) in the model state dict, and map it

Samples the logits to generate outputs Given the logits of the last token, this function samples it according to the parameters defin

Method to set the operating mode

Returns the local shard of the given tensor if it is a DTensor.

Build an attention-mask having shape (bs, 1, q_len, kv_len) from source_block and target_block Args: source_block (torch.

Forward method with activation checkpointing.

args: y_true : 3-d ndarray in [batch_size, img_rows, img_cols] y_pred : 3-d ndarray in [batch_size, img_rows, img_cols] retur

Get the index offset of current pipeline stage, given the level of pipelining.

Track the rng state.

Pseudo-deallocate (i.e., set to scalar) the output tensor's '.data' field. This method should be called right after the output tensor has been

Exchanges local metadata with `all_gather_object` to determine global metadata. Args: sharded_state_dict (ShardedStateDict): local sharde

Run inference and post-process outputs, i.e., detokenize, move to cpu and convert to list. Args: data_parallel (bool): Enable data pa

Divide existing and missing blocks evenly across all ranks. See 'get_blocks()' above for description. The returned lists of existing and miss

Get optional tensor arguments for CUDA graph.

Return caller's rank in the data-parallel group.