Get the tensor associated with the given cache ID. Parameters: cache_id (int): The ID of the cache tensor to get.

Return a mock object that satisfies dist.get_world_size / get_rank.

Compile the module using the specified backend and kwargs. If a compiler_fn is set, it will be used instead of torch.compile().

Returns the pre-allocated, padded output Tensor.

Collect gradients from model parameters using safe_get_full_grad API

Return torch.ops.deepspeed.<name>, raising clearly if not registered.

Compare gradients between DDP and DeepSpeed. Uses PyTorch's default tolerances for the tensor dtype (e.g., for bfloat16: rtol=1.6e-2, atol=1e

Collect gradients from DDP model

Given a DSModule key, attempt to instantiate

Overrides the internal iteration index

Advances an iterator along logical dimensions of matrix in units of whole tiles

Assemble and return the fp32 gradient of a low-precision (e.g., fp16) parameter. The return data type is that used for gradient accum

Validate config constraints. Raises ValueError on invalid config.

Build the immutable per-stage folding spec from public config sizes.

Returns a pointer

Run a small CPU/Gloo distributed test without requiring visible GPU devices.

Returns whether access is valid or not

fp32 config toggling managed_gradient_accumulation; micro-batch 1 so total_samples == micro-batch count.

Checkpoint a model or part of the model. This has been directly copied from torch.utils.checkpoint. Tensor arguments passed by keyword partic

r"""Coalesce ZeRO 1/2/3 gradient reduction across multiple engine.backward() calls. One with-block == one optimizer step: every backward insid

Manually overrides the DeepSpeed engine's gradient accumulation boundary state, this is an optional feature and should be used with c

Extract tp_plan from HuggingFace model. Merge plans from the model config, model class, and runtime instance. HuggingFace may replace the ins

< Efficiently disables all accesses guarded by mask