A residual block that can optionally change the number of channels. :param channels: the number of input channels. :param emb_channels: t

A sequential module that passes timestep embeddings to the children that support it as an extra input.

An attention block that allows spatial positions to attend to each other. Originally ported from here, but adapted to the N-d case. https

A single linear layer which does a 1x1 conv

A downsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determini

Transformer block for image-like data. First, project the input (aka embedding) and reshape to b, t, d. Then apply standard transform

An upsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determinin

A utility attention mask class that allows one to: - Create a causal 4d mask - Create a causal 4d mask with slided window

A module which performs QKV attention and splits in a different order.

Adapted from CLIP: https://github.com/openai/CLIP/blob/main/clip/model.py

Parallel cross-attention layer for Transformer

to match AttnBlock usage

A convolutional layer that can be used with LoRA.

A Linear layer that can be used with LoRA.

Uses xformers efficient implementation, see https://github.com/MatthieuTPHR/diffusers/blob/d80b531ff8060ec1ea982b65a1b8df70f73aa67c/src/diffu

Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MistralDecoderLayer`] Args: config: MistralCo

MLP as used in Vision Transformer, MLP-Mixer and related networks

A module which performs QKV attention. Matches legacy QKVAttention + input/ouput heads shaping