Update the CUDA sync point if the priority of the new point is higher than the current sync point, which means the new point is reach

Use multi-tensor-applier to copy values from one list to another. We don't have a bfloat16 implementation so for now if the overflow_

Helper function to notify condition variable when a new request is added.

Preprocesses inputs for the transformer decoder. Applies embeddings to input tokens, or uses `decoder_input` from a previous pipeline

Prompts the user, takes care of /h (followed by requery) and sets the mode. Returns the user input.

Reduce-scatter the input tensor across model parallel group. Args: input_ (torch.Tensor): The input tensor to be reduce-scattered.

Set the main param and optimizer states corresponding to the input model_param. The structure of the input `tensors`: tensors = {

Split an aggregated tensor into multiple tensors at the batch dimension. Args: aggregated_tensor: The tensor to split

Update the display based on current mode.

Try to acquire the lock. Returns True if acquired, False otherwise. Caller should retry until it returns True.

Switch from single-line to multi-line mode (one-way only).

Turn ShardedTensorFactories into ShardedTensors *in-place*. Args: sharded_state_dict (ShardedStateDict): state dict possibly

Apply merges defined by ShardedTensorFactories *in-place*. Args: x1 (StateDict): state dict loaded from the checkpoint x2 (Sharde

Backward pass for DownProjFunction - Pure Python implementation. Forward: output = topk_weights * (input @ w2) (with topk_weight mul

Compute weight gradients during the backward pass if delay_wgrad_compute is enabled.

Build dataloader given an input dataset.

Calculate log probs for all active requests and return them. TODO: @wdykas support top-n log probs. Args: logits (Tensor

Combines expert outputs via communication and adds shared expert output. This method uses the token dispatcher to combine the outputs from di

Merged forward and backward step for combined 1f1b scheduler. Args: Need to accept the argument of both forward_step() and backward_step(

Compute KL divergence loss between index_scores and true attention_scores. This loss trains the indexer to predict which tokens are importan

Compute the total number of pipeline stages across a multi-module chain. Interprets ``topology`` as a directed acyclic graph (DAG) where node

Concat with special tokens and pad sequence to self.max_seq_length

Check if a sub-mesh exists in the device mesh by name.

Converts a sequence of [tokens|ids] using the vocab.

Should be called at the end of each schedule function, (e.g. forward_backward_pipelining_with_interleaving) in `megatron.core.pipeline_paralle

Capture CUDA Graphs per TransformerLayer per microbatch.

Create placement groups for actor and rollout engines.

Cumulative query sequence lengths.

Convert list of token IDs to plaintext.

Deserialize request. Args: obj (dict): Serialized request data. Returns: (InferenceRequest) Deserialized req

Dispatches tokens to assigned expert ranks via communication. This method performs the actual communication (e.g., All-to-All) to distribute

Embed a text dataset. Args: text_dataset (torch.utils.data.Dataset): Text dataset to embed. Each sample of the te

Enable the feature flag.

Enable forward pre-hooks needed for param all-gather overlap with forward compute.

Convert text to a list of token IDs.

Helper function to evaluate and dump results on screen.

Score a single (question, response) pair against a preference. Calls GPT-4.1 `num_votes` times in parallel and returns the average of al

Find the longest cached prefix for the given text. Args: text: Input string to find prefix for Returns:

Set the last layer flag.

Finishes grad sync (all-reduce or reduce-scatter) communication operations for all model gradients. When overlap_grad_reduce

Fix up query/key/value matrix ordering if checkpoint version is smaller than 2.0

Forward.

wrap forward_helper, backward_helper, and combined_forward_backward_helper in a unified way

Check if a FP8 tensor needs transpose data.

The megatron core inference backend generate function This uses dynamic engine if available, otherwise uses legacy static engine. Ar

Returns the current number of active requests.

Get directory of saved encodings. Args: config (RetroPreprocessingConfig): Retro preprocessing config. Returns: Path to the

Get the alignment size for quantization.

Constructs an attention mask given the input sequence length.

Get TP-CP ranks at boundary PP stage for each DP replica. Returns ranks at the last PP stage (if src) or first PP stage (if dest) for

Get or create a buffer for all-to-all communication. Args: group (torch.distributed.ProcessGroup): Process group for communication

Get the path to a config file.

Returns a slice of the conv1d bias relevant to the current context parallel rank

Returns a slice of the conv1d weight relevant to the current context parallel rank

Get current running global batch size. If decrease_batch_size_if_needed is False, this just equals global batch size.

Get the data-parallel group the caller rank belongs to.

Get DB chunk as text.

Get file path to empty index (i.e., trained, but unpopulated). Args: config (RetroPreprocessingConfig): Retro preprocessing confi

Return the eos token id. We insert eos_token between two samples during packing. However, if the eos_token is used in message or after turns,

Return fp4 context manager.

Get the fp8 context for the transformer layer.

Mix the native spec with TENorm. This is essentially the native local spec except for the layernorm implementation is using TENorm from Trans

Calculate the norm of gradients in fp32. This is adapted from torch.nn.utils.clip_grad.clip_grad_norm_ and added functionality to handle mode

Calculate the number of hidden bytes for a tensor. Args: x (torch.Tensor): Input tensor Returns: int: Number of hidden bytes