Get the data parallel group the caller rank belongs to.

Returns list of include paths, relative to root of SwissArmyTransformer package (i.e., SwissArmyTransformer/sat)

Initialize torch.distributed.

in config: "target": module type, vqvae_zc.Decoder/vqvae_diffusion.Decoder/vqvae_diffusion.Decoder2 "ckpt": path of c

Returns list of source files for your op, relative to root of sat package (i.e., SwissArmyTransformer/sat)

Zero out the parameters of a module and return it.

convert Ids to tokens accounting for command tokens, tokens are joined and returned as a string.

Initialize affine weight for model parallel. Build the master weight on all processes and scatter the relevant chunk.

Returns absolute build path for cases where the op is pre-installed, e.g., sat.ops.adam.cpu_adam will be installed as something like:

Evaluate a function without caching intermediate activations, allowing for reduced memory at the expense of extra compute in the backward pas

in-place copy from src to dst src and dst should be the same layer type, e.g., both are LayerNorm or both are Linear. Or at least, bo

Returns optional list of compiler flags to forward to the build

embed_dim: output dimension for each position pos: a list of positions to be encoded: size (M,) out: (M, D)

List all available args of the current model.

Reset timer.