Layer Norm, fused into a single CUDA kernel. Borrow from: https://github.com/NVIDIA/Megatron-LM/blob/6501752396e9cc360ce894cda4b2217a58c1c09d

Embeds class labels into vector representations. Also handles label dropout for classifier-free guidance.

CustomMLP will take the input with h hidden state, project it to 4*h hidden dimension, perform nonlinear transformation, and project the

A lightweight class for recording time without any distributed barrier. This class allows for recording elapsed time between events without

The final linear layer of DiT.

Inference parameters that are passed to the main model in order to efficienly calculate and store the context during inference.

Rotary position embedding w/ concatenatd sin & cos The following impl/resources were referenced for this impl: * https://github.com/lucidrain

Image to Patch Embedding

Embeds scalar timesteps into vector representations.

Transformer class.

A single transformer layer. Transformer layer takes input with size [s, b, h] and returns an output of the same size.

Vision Transformer with support for patch or hybrid CNN input stage

Vision Transformer with support for patch or hybrid CNN input stage

VideoDiT model for video diffusion. Args: config (MagiConfig): Transformer config pre_process (bool, optional): Include embedding

Attention layer abstract class.

r""" :class:`~models.AutoModel` is a generic model class that will be instantiated as one of the base model classes of the library when cr

Singleton metaclass

Abstract base class for video tokenizers. This class defines the interface for video tokenizers and provides common methods and properties.