Return generator over underlying items.

Get the size of the dtype/code in bytes Args: key (Union[int, Type[numpy.number]]): The dtype or code Raises:

Remove and set the next non-empty chunk as the current backward chunk.

Set a key in the global config file.

If distributed is initialized or rank is specified, print only on rank 0.

Return arguments.

Return whether the IndexedDataset exists on disk at the prefix Args: path_prefix (str): The prefix to the index (.idx) and data (

Get an item from the queue with Python < 3.13 compatibility.

Updates internal state in grad scaler based on whether NaNs are seen in grads or not.

Initialize model parallel cuda seed. This function should be called after the model parallel is initialized. Also, no torch.cuda.manual_seed

Return a list of information on the exceptions that the function may raise.

Checks if the internal RNG state has been set wirth set_states().

Write timers to a tensorboard writer. Note that we only report maximum time across ranks to tensorboard. Args: names (Lis

Load the sharded part of the checkpoint.

Executes a python command on the server. It can be used to execute the pyautogui commands, or... any other python command. who knows?

Allows to construct the ShardedTensor given offset specified in process ranks. Args: key (str): unique key data (torc

Check if minimum version of `transformer-engine` is installed.

If torch distributed is initialized, write log on only one rank Args: logger (logging.Logger): The logger to write the logs args

Build a module from a ModuleSpec or return it as is if already provided.

Handles a POST request.

Instantiate a PreTrainedBertModel from a pre-trained model file. Download and cache the pre-trained model file if needed.

Use this spec to use lower-level Transformer Engine modules (required for fp8 training). Args: num_experts (int, optional): Number of ex

r"""Create a process group based on a list of dimension names Note: The unique key used to store the process group internally will follow the

logs the timers passed in names to stdout. Example usage is to log average per step value for timer 'foo', this function can be called with

Orchestrate the entire flow of loading the Megatron checkpoint.

Wait for all gather operations to complete

Convert a conversation to token IDs.

Adds a new value to the stream

Saving entrypoint. Extracts ShardedTensors from the given state dict. Rank 0 saves the "regular" part of the checkpoint to common torch file.

Check if minimum version of `transformer-engine` is installed.

Return tokenizer.

Compute sum of per-sample means across variable-length responses. Parameters: x: Flat tensor containing concatenated per-token values acr

Start the timer. Args: barrier (bool, optional): Synchronizes ranks before starting. Defaults to False.

Clip gradients (if needed) and step the base optimizer. Always return successful since there is no overflow.

Loading entrypoint. In the steps below, the following verbs refer to corresponding objects: - load = load from checkpoint - extract = ext

All gather the input tensor to the output tensor

Match a layer based on its qualified name. If it does not match, return None. If it matches, return the configuration key to

Check if minimum version of `torch` is installed.

Log on rank 0. Args: message (str): Message to log.

Issue a deprecation warning if tp_group is None and return the default tp group.

Save a model, optimizer and optionally dataloader checkpoint. Checkpointing context is used to persist some checkpointing state throughout a

Get rank for a distributed group. Args: group: Process group to get rank for. If None, uses default group. Returns: int: Ran

Use the default process groups from parallel_state. Args: required_pgs (List[str], optional): List of process group name

Fork the cuda rng state, perform operations, and exit with the original state.

Use this spec for an implementation using only modules in Megatron-Core. Args: num_experts (int, optional): Number of experts. Defaults

Get rank group by input token. Args: token (str): Specify the ranks type that want to get. If we want

Load a model checkpoint and return the iteration. strict (bool): whether to strictly enforce that the keys in :attr:`state_dict` of the ch

Unwrap_model to return the final model instance

Connects to the coordinator and starts the background listener task. This method must be awaited before submitting any requests. It

Create a HyperCommGrid with tensor parallelism=2, context parallelism=2, and data parallelism=2.

Record a new sample.

Stop the timer. Args: barrier (bool, optional): Synchronizes ranks before stopping. Defaults to False.

Pause health checking. Called when engines are offloaded.

Get world size for a distributed group. Args: group: Process group to get world size for. If None, uses default group. Returns:

Convert model weights to trtllm model weights This method goes through each layer in the model state dict and converts to equivalent trtllm m

Test fp8_param with gpt_model.

Builds a ShardedStateDict from the original tensor

Extract all text content from the app's UI.

Stops the client and cleans up all resources. This method cancels the background listener task, closes the ZMQ socket, and t

Print debug message for a specific rank when DEBUG is enabled.

Compare two values with a tolerance of 0.1% (0.005)