We use two heuristics to compute the batch size 1. We use the Lowest Common Multiple of the micro-batches as the base batch size and scale

Return world size for the data parallel group.

Generate expert parallel and expert data parallel group ranks list. Example - E + M + D parallel world_size = 16 model_degree

Get expert weight tensor by name, handling both attribute and child module patterns.

Return the conversion-facing view of AutoTP UC metadata for a parameter. AutoTP keeps a single parameter-level metadata object with two roles:

Return the restore-facing view of AutoTP UC metadata for a parameter. AutoTP parameter metadata intentionally serves two separate consumers:

the group id is always the smaller rank unless its a wrap around

Per-parameter selective update for offloaded partitions (CPU or NVMe). NVMe partitions are swapped in, updated, and swapped back out one at a

Check if module stores expert weights as 3D parameter tensors (transformers 5.0.0+). Returns True if the module has a parameter named preset.expe

Returns whether or not all parameters have been initialized and transformed by the model. Once this returns True, all the `ParameterB

Merge-base of ``base`` and ``HEAD``, or "" if it can't be determined. An empty result (e.g. shallow clone that doesn't reach the fork point)

Dotted module name for a file under a package root, or None. e.g. deepspeed/runtime/engine.py -> deepspeed.runtime.engine tests/

Parse every file once; return (deps_by_file, raw_imports_by_file).

Returns the prime factorization of positive integer N.

Process gradients for selected columns in FP32 groups Args: param: The parameter to process param_id: ID of

All-reduce the input tensor across model parallel group.

Reduce (sum) a new local result into the existing local results. Subclasses should override this method. For a given ``out_id``, thi