Returns an iterator over optimizer parameters

Performs a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model

Returns a dict containing the current state of this :class:`FP16_Optimizer` instance. This dict contains attributes of :class:`FP16_O

Performs a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model

Get optimizer parameter Can either accept two ints or one DistributedFusedAdam.ParameterFragment. Arguments: par

TODO: better name

Initialize optimizer state for parameters Ignores parameters that have already been initialized. Arguments: params (iter

Datatypes for the bucket's compute and communication

Generate CUDA Stream name from stream index number. Args: stream_idx: Non-negative index number. 0 refers to the default stream, others r

Make sure PyTorch and Apex gradient clipping produce same results

Copy between corresponding buffers Uses fused copy kernel if possible.

Ensure that all gradients are synchronized

Call this method to modify your model to take advantage of sparse matrix multiplication. Note that this call alone only augments the model wit

Call this method to monkey patch optimizer step function so that masks can be applied to gradients and weights during training. You mu

Whether two PyTorch devices are equivalent

Call this method to enable sparsity. If init(...) was called with allow_recompute_mask=False AND sparsity is disabled, pruned field can be Non

Get the CuDNN front-end context manager. Returns: CuDNNManager: Global CuDNN manager.

It is expected that all threads in the CTA enter this function!

Construct parameters with random configurations

Ensure that all parameters are synchronized

This function is used to save the graph into JSON file for inspection.

Test state_dict and load_state_dict functions Two models are constructed, possibly on different process groups. One of the models is

Compute and apply scaling factor for scaled optimizer state The scaling factor is chosen to maximize the dynamic range while avoiding

Make sure local shards of parameters are in expected datatypes The Adam kernel only supports floating-point datatypes. If we want to

Add asynchronous request to coalescing manager

Wait for any gradient synchronizations that are in progress

Allocate contiguous buffer for grad buckets

Initialize optimizer state for a parameter

Synchronize parameter buckets Parameter synchronization is asynchronous. Involves all-gather over distributed process group. Assumes

Generate data structure for recording steam on return tensors before program exit. Args: buffers: Names of buffers that need to b