If distributed is initialized, print only on rank 0.

Return a string describing the data distribution.

Process batch and produce inputs for the model.

Process batch and produce inputs for the model.

Process batch and produce inputs for the model.

Process batch and produce inputs for the model.

Process batch and produce inputs for the model.

Read prompt data

Creates a final sharded state dictionary from a tensor-aware state dictionary. Args: sharded_state_dict (ShardedStateDict): T

Receive tensor from previous rank in pipeline (forward receive). See _communicate for argument details.

Receive from previous pipeline stage and update the input buffer inplace.

Registers grads for the passed-in param to be "ready" for grad sync. When the number of microbatches is greater than 1, we only want

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Simple GPU memory report.

Whether a given strict flag involves mismatch check against the checkpoint.

Zero out the underlying grad_buffer.

Restores the original state dict from a reformulated form. Inverse of `apply_nd_flattened_tensors_reformulation`. Args: state_dict (

Run inference and post-process outputs, i.e., detokenize, move to cpu and convert to list.

Method to rewind the data iterator to the first microbatch of the iteration.

Rotates half the hidden dims of the input.

Rotates half the hidden dims of the input.

Rotates half the hidden dims of the input.

Rotates half the hidden dims of the input.

Safely set tensor's '.data' field. Check first that the tensor is viewless (i.e., '._base' not set). If not, raise an exception.

Sample and generate a token. Note: logits has the dimension [b, v] where b is the batch size and v is the vocabulary size. If vocab

Sample a random subset of all blocks. Args: _blocks (List[Optional[Dict]]): List of all blocks. Returns:

Preprocesses the given state dictionary by applying factories, discarding non-persistent data and extracting the common state dictionary. Opti

Function for just scoring. Args: model: no interleaving is supported. tokens: prompt tokens extended to be of size [b, max_prompt

Wrapper for p2p_communication.send_backward used with non-interleaving schedule.

Send tensor to next rank in pipeline (forward send). See _communicate for argument details.

Wrapper for p2p_communication.send_forward used with non-interleaving schedule.

Send output to the next pipeline stage.

Set the document indices This method is slated for deprecation. Args: document_indices (numpy.ndarray): The document ind

See megatron.model.transformer.set_input_tensor()

Set the layer number for the router.

Set shared expert to the dispatcher.

Generate a sharded state dictionary for the transformer block. Args: prefix (str, optional): Prefix to be added to all k

Maps local expert to global experts. The sharded_state_dict for the weight parts are compatible with the SequentialMLP, where

Method instructing whether to checkpoint and/or abort the job. Args: None Returns: A tuple formed of:

Method instructing whether to (re)run the forward-backward pass. Args: data_iterator: data iterator or list of data iterators use

Return just one element from each dimension (0, 0, 0, ...). If you prefer to keep the entire hidden dimension and only slice the sequence

Initiates grad sync (all-reduce or reduce-scatter) communication operations for all buckets in the bucket group. When ddp_co

Save dict with state dicts of each of the models.

For easy load when model is combined with other heads, add an extra key.

Save dict with state dicts of each of the models.

Clip gradients (if needed) and step the base optimizer. Always return successful since there is no overflow.

Step the optimizer with ready gradients, return successful.

Calculate position ids from token ids Args: token_ids (Tensor): input tokens Returns: Tensor: position ids

Convert text to embedding ids Args: text (str): The text to convert Returns: numpy.ndarray: The converted em

Tokenize prompts and make them avaiable on all ranks. Args: data_parallel (bool): Broadcast tokens across a single data parallel model re

Track the MoE metrics for logging.

Entry point for training the index. We select whether to train a new index, or validate an existing index. Args: config (RetroPrepro

Validate added encodings. Steps: - Randomly sample subset of text dataset blocks. - Encode each block. - Compare against saved encodi

Validate training embeddings. Steps: - Randomly sample subset of text dataset blocks. - Embed each block. - Compare against saved emb

Performs cross entropy loss when logits are split across tensor parallel ranks Args: vocab_parallel_logits: logits split across tens

Vocab range from per partition vocab size.

Args: x: (B, H, W, C) window_size (int): window size Returns: windows: (num_windows*B, window_size, window_size, C)

Args: x: (B, H, W, C) window_size (int): window size Returns: windows: (num_windows*B, window_size, window_size, C)

Write arguments to tensorboard.

Encourages the router's logits to remain small to enhance stability. Please refer to the ST-MoE paper (https://arxiv.org/pdf/2202.08906.pdf) for d

Zeros out all grad buffers. Needs to be called at the beginning of each training iteration.

Zero out all parameters in embedding.

Calls the forward pass of the cudagraphed module. Args: megatron_module (torch.nn.module): The megatron module to be graphed and

Initialize Timer. Args: name (str): Name of the timer.