Clear experiment queues, does not reset self.experiment_count

Recursively get the attribute of a module. Args: model (`torch.nn.Module`) The model to get the attribute from. m

Set random seed for reproducability.

given a generator of metrics for each of the data points X_i, write the metrics, text, and labels to a csv file

Get the model parallel group the caller rank belongs to.

Return my rank for the tensor model parallel group.

True if this process is in the last stage in the pipeline.

Return my rank for the model parallel group.

Return the number of processes along the given axis. For example: >>> X = ProcessTopology(axes=['x', 'y'], dims=[2,3])

Return world size for the model parallel group.

Query whether the current micro-batch is at the boundary of gradient accumulation, and thus will trigger gradient reductions and

Ensure that numerator is divisible by the denominator and return the division value.

Return timers.

Communicate tensors between stages. Used as helper method in other communication methods that are used in megatron/schedules.py. Arguments:

Track the rng state.

r

Return my rank for the pipeline model parallel group.

encode text using text tokenizer and shift Id values for command tokens

Pass through attributes defined in the model if they are not overridden by ds-engine.

Backwards-compatible way of querying the tensor model parallel rank from an ``mpu`` object. *Tensor* model parallelism means that tensors are

Remove new lines and multiple spaces and adjust end of sentence dot.

Return the coordinate owned by a process rank. The axes of the returned namedtuple can be directly accessed as members. For example:

Return the global rank that follows the caller in the pipeline

Return the global rank that preceeds the caller in the pipeline

Return world size for the pipeline model parallel group.

decorator that causes an NVTX range to be recorded for the duration of the function call.

Log a group of timers.