This method should create a generator of tuples of the form (name, parameter) for all parameters in the model. The name should be the

Create the inference parameter.

Retrieve and materialize stored real inputs Returns: Materialized real inputs with actual tensors Raises:

Merge partitioned weights from flat_groups into a single tensor.

Return the global rank of a process via its coordinates. Coordinates are specified as kwargs. For example: >>> X = ProcessTopolo

Retrieves a single item from the dataset with the option to only return a portion of the item. get(idx) is the same as [idx] but get

Update the tuner with what configurations have been evaluated and their performance results

Return torch.ops.deepspeed.<name>, raising clearly if not registered.

Save training checkpoint Arguments: save_dir: Required. Directory for saving the checkpoint tag: Optional. Checkpoint

Log message when get_dist_msg() deems it should be logged, see its docstring for details. Args: message (str) ranks (list)

Load training checkpoint Arguments: load_dir: Required. Directory to load the checkpoint from tag: Checkpoin

Track the rng state.

Returns whether a module in `model` has been flagged as a 'leaf' module. See `set_z3_leaf_modules` for more details. Args: