Create layout tensor for the given sequence length Arguments: seq_len: required: an integer determining number of attention head

Utility for sharding a parameter. This will return the slice of the parameter that should exist on the given shard_rank given the sharding co

Split parameters into different MoE groups for optimizer Args: param_groups (Union[Dict[str, Any], Tuple[Dict[str, Any], ...], List[Dict[

Stores a fragment to memory

Mapping of token to which sequence it belongs to in the ragged batch. If the device Tensor is requested, the Tensor is truncated to t

Implements TopKGating on logits.

Updated the running absmax used to calculate params. Function Arguments : val : The __half2 value to update the running min and max with.

Acquire process-wide ZeRO-3 Dynamo config ownership and return its release callback.

Cast params to param_dtype; cast buffers only when buffer_dtype is set.

Resolve (param_dtype, buffer_dtype) for the module cast. param_dtype follows the enabled fp16/bf16 mode (None if neither). buffer_dty

Helper for embedding test to limit the number of tests to run. Params: embed_dim (int): Model dimension vocab_size (int): Le

Helper for `release_[component]` methods. Accepts a list of tuples where the first element is the module param that needs to be delet

r"""Functional API that performs AdamW algorithm computation. See :class:`~torch.optim.AdamW` for details.

Adds a pointer offset in units of Element

Backwards-compatible way of querying the pipeline parallel world size.

If all heads require same sparsity layout, it propagate first head layout to all heads Arguments: layout: required: a tensor of

< Efficiently disables all accesses guarded by mask

Compute AllToAllV split sizes for token dispatch/combine. ``num_tokens_per_expert`` may be supplied by the caller to reuse the histogram alre

The number of pipelines.

Get the sequence descriptor for the given sequence id. If the sequence does not exist, then None is returned.

Initialize the DeepSpeed Engine. Arguments: args: an object containing local_rank and deepspeed_config fields. This is option

True if this process is in the first stage in the pipeline.