Elasticity configuration error

A wrapper for integers that will return a custom string or comma-formatted string of the integer. For example, print(pp_int(1e5)) will return

Tokenization object to hold tokenization, (processed text),and original text. Can hold tokenization as Ids or tokens. It also holds

Initialize the DeepSpeed GPT Transformer Layer.

Sets parameters for DeepSpeed Inference Engine.

Group of timers. Borrowed from Nvidia Megatron code

Wrapper for the Transformer-Engine's `Linear` layer but specialized similar to megatron's `ColumnParallelLinear` layer.

Elastic config object, constructed from a param dictionary that only contains elastic config parameters, example below: If elasticity is

Base exception for all elasticity related errors

Compute a forward pass. Roughly: .. code-block:: python buffers['outputs'][buffer_id] = forward(buffers['inputs'][buffer_id])

GPT-2 Language model.

Load a micro-batch into a buffer. Roughly: .. code-block:: python buffers['inputs'][buffer_id] = next(data_iter)

MLP. MLP will take the input with h hidden state, project it to 4*h hidden dimension, perform nonlinear transformation, and project the s

Parallel self-attention layer abstract class. Self-attention layer takes input with size [b, s, h] and returns output of the same size.

Errors related to the use of deepspeed.PipelineModule

Receive activations from the previous stage in the pipeline. Roughly: .. code-block:: python buffers['inputs'][buffer_id] = recv()

Send activations to the next stage in the pipeline. Roughly: .. code-block:: python send(buffers['outputs'][buffer_id]) .. not

Abstract Configuration class to store `sparsity configuration of a self attention layer`. It contains shared property of different block-sparse sp

Compute a backward pass and accumulate gradients. Roughly: .. code-block:: python outputs = buffers['outputs'][buffer_id] g

Serializable data structure for holding data for blocks -- embeddings and necessary metadata for REALM

Checks for overflow in gradient across parallel process

Initialize the DeepSpeed BERT Transformer Layer.

Initialize the DeepSpeed Transformer Config. Arguments: hidden_size: The hidden size of the transformer layer intermed

Initialize the DeepSpeed Megatron GPT Transformer Layer.

Initialize the DeepSpeed Transformer Config. Arguments: batch_size: The maximum batch size used for running the kernel on each GP

Similar to normal implementation of distributed sampler, except implementation is at the batch sampler level, instead of just the sampler leve

Dummy optimizer presents model parameters as a param group, this is primarily used to allow ZeRO-3 without an optimizer

Class that manages dynamic loss scaling. It is recommended to use :class:`DynamicLossScaler` indirectly, by supplying ``dynamic_loss_scale

Class that manages dynamic loss scaling. It is recommended to use :class:`DynamicLossScaler` indirectly, by supplying ``dynamic_loss_scale=T

FP16 Optimizer for training fp16 models. Handles loss scaling. For usage example please see, TODO: DeepSpeed V2 Tutorial

Measures the latency, number of estimated floating-point operations and parameters of each module in a PyTorch model. The flops-profiler profiles

Dataset containing sentences and their blocks for an inverse cloze task.

BERT-based encoder for queries or blocks used for learned information retrieval.

Loader for IndexedDataset

Loader for IndexedDataset

Block-Sparse MatMul class; this class handles three types of matrix-multiplication: - sparse = dense X dense - dense = sparse X dense

This handle assumes that no need to copy data out from a contiguous tensor

Performs one step with the optimizer and zeros gradients. .. note:: Should be issued after :class:`ReduceGrads` and :class:`ReduceTiedGrads`.

A topology specialization for hybrid data and pipeline parallelism. Uses data parallelism on the last dimension to encourage gradient

Class to quantize given activations. Note that when using this function, the input activation quantization range will be fixed for all tokens

Reduce the computed gradients among data-parallel processes within the stage.

Wrapper for the Transformer-Engine's `LayerNorm`.

Wrapper for the Transformer-Engine's `Linear` layer but specialized similar to megatron's `RowParallelLinear` layer.

Group of timers.

Tokenizer object that handles text tokenization, command tokens, and type tokens. Command tokens and text tokens are stored together in o

Gate module which implements Top2Gating as described in Gshard_. :: gate = TopKGate(model_dim, num_experts) l_aux, combine_weight

A light-weight wrapper class for torch.distributed API. Only a subset of functions are wrapped. Once the init_process_group i

Original BERT wordpiece tokenizer.

Anneals the learning rate.

The DeepSpeed Autotuner automatically discovers the optimal DeepSpeed configuration that delivers good training speed. The Autotuner uses model inform