Create a linear module.

Create a 1D, 2D, or 3D convolution module.

Make a standard normalization layer. :param channels: number of input channels. :return: an nn.Module for normalization.

Transpose a 3D tensor [b, s, np*hn] into a 4D tensor with size [b, np, s, hn].

Create sinusoidal timestep embeddings. :param timesteps: a 1-D Tensor of N indices, one per batch element. These may be

Evaluate a function without caching intermediate activations, allowing for reduced memory at the expense of extra compute in the backward pas

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Load a pretrained checkpoint of the current model. Args: name: The identifier of the pretrained model. arg

embed_dim: output dimension for each position pos: a list of positions to be encoded: size (M,) out: (M, D)

Zero out the parameters of a module and return it.

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T