Helper function for validating the GEMM kernel for a single expert.

Returns a TensorRef to the B operand

Return the AutoEP preset name for a supported HF model_type.

Prints the model graph with the measured profile attached to each module. Args: profile_step (int, optional): The global training

This is an ugly hack to get the unittest suites to play nicely with pytest. Otherwise failed tests are not reported by pytest for some reason.

Re-enter backward context for subsequent phases in reentrant checkpointing.

Replaces values in dict d with values in dict u. Args: d (dict): the target dict to overwrite u (dict): the dict containing the v

Replace bert-style transformer layers with DeepSpeed's transformer layer Arguments: orig_layer_impl (torch.nn.Module): the original trans

Get the local partition of a ZeRO-3 partitioned parameter in fp32 precision. Args: param (``torch.nn.Parameter``): A model parame

Get the local gradient partition of a ZeRO-3 partitioned parameter. The return data type is that used for gradient accumulation. This

Get the local optimizer state partition of ZeRO-3 partitioned parameter in fp32 precision. Args: param (``torch.nn.Parameter``):

Scales the loss by the current loss scale. We need this function to scale loss without calling backward on it.

Set the DEEPSPEED_AUTOTP_MODE based on the training flag

Store an iterator to sample for training data.

Sets a leaf flag for modules referenced by their names in ``model.named_modules()``. Args: model (torch.nn.Module): The model cont

Sets a leaf flag for modules referenced by suffixes of ``model.named_modules()`` names.

Args: max_log_level_str: maximum log level as a string Returns ``True`` if the current log_level is less or equal to the specified l

Return True when count_used_parameters_in_backward() should be re-evaluated.

Starts profiling. Extra attributes are added recursively to all the modules and the profiled torch.nn.functionals are monkey patched.

Update the model parameters. .. note:: This method will be called internally by ZeRO-Offload. DeepSpeed users should

Store a fragment to memory

Returns a torch.LongTensor that encodes partitioning information. Can be used along with ``data()`` to serialize a ``PartitionedTensor`` for

Compare two tensors or non-tensor numbers for their equality. Add msg=blah to add an additional comment to when assert fails.

r

Perform any necessary transformations of the gate parameter. Args: param (torch.Tensor): gate_w (shape: [num_experts, mo

Shards the first MLP parameter and passes it to the underlying implementation for any necessary transformations. Args:

Compute the trimmed mean of a list of numbers. Args: data (list): List of numbers. trim_percent (float): Percentage of data to tr

Return a preset carrying an actionable diagnostic for an unsupported HF model_type.

Update hook state after a gradient hook fires and run epilogue if all hooks have fired.

Zero FP16 parameter grads.