This method should create a generator of tuples of the form (name, parameter) for all parameters in the model. The name should be the

Create the inference parameter.

Retrieve and materialize stored real inputs Returns: Materialized real inputs with actual tensors Raises:

Merge partitioned weights from flat_groups into a single tensor.

Return the global rank of a process via its coordinates. Coordinates are specified as kwargs. For example: >>> X = ProcessTopolo

Retrieves a single item from the dataset with the option to only return a portion of the item. get(idx) is the same as [idx] but get

Update the tuner with what configurations have been evaluated and their performance results

Return torch.ops.deepspeed.<name>, raising clearly if not registered.

Save training checkpoint Arguments: save_dir: Required. Directory for saving the checkpoint tag: Optional. Checkpoint

Load training checkpoint Arguments: load_dir: Required. Directory to load the checkpoint from tag: Checkpoin

Log message when get_dist_msg() deems it should be logged, see its docstring for details. Args: message (str) ranks (list)

Returns the pre-allocated, padded output Tensor.

Copy residency state, optionally starting a candidate-local peak.

Compile the module using the specified backend and kwargs. If a compiler_fn is set, it will be used instead of torch.compile().

Track the rng state.

Assemble and return the fp32 gradient of a low-precision (e.g., fp16) parameter. The return data type is that used for gradient accum