Wrapper class of torch.nn.Linear Weight initialize by xavier initialization and bias initialize to zeros.

Residual Connection Module. outputs = (module(inputs) x module_factor + inputs x input_factor)

Conformer Feed Forward Module follow pre-norm residual units and apply layer normalization within the residual unit and on the input before t

Conformer encoder first processes the input with a convolution subsampling layer and then with a number of conformer blocks. Args:

Extension of Sampler that will do the following: 1. Change the batch size (essentially number of sequences) in a batch to ensure that the

Conformer convolution module starts with a pointwise convolution and a gated linear unit (GLU). This is followed by a single 1-D depthwise co

Conformer employ multi-headed self-attention (MHSA) while integrating an important technique from Transformer-XL, the relative sinusoidal pos

When kernel size == 1 conv1d, this operation is termed in literature as pointwise convolution. This operation often used to match dimensions.

Swish is a smooth, non-monotonic function that consistently matches or outperforms ReLU on deep networks applied to a variety of challenging

Conformer block contains two Feed Forward modules sandwiching the Multi-Headed Self-Attention module and the Convolution module. This sandwic

Conformer block contains two Feed Forward modules sandwiching the Multi-Headed Self-Attention module and the Convolution module. This sandwic

Conformer encoder first processes the input with a convolution subsampling layer and then with a number of conformer blocks. Args:

Convolutional 2D subsampling (to 1/4 length) Args: in_channels (int): Number of channels in the input image out_channels (in

Convolutional 2D subsampling (to 1/4 length) Args: in_channels (int): Number of channels in the input image out_channels (in

When groups == in_channels and out_channels == K * in_channels, where K is a positive integer, this operation is termed in literature as dept

The gating mechanism is called Gated Linear Units (GLU), which was first introduced for natural language processing in the paper “Language Mo

r""" modified from diffusers/src/diffusers/models/attention.py notes. _c: context related. text, cond, etc. (left part in sd3 fig2.b)

Multi-headed attention. See "Attention Is All You Need" for more details.

Relative positional encoding module. Args: d_model: Embedding dimension. max_len: Maximum input length.

Multi-head attention with relative positional encoding. This concept was proposed in the "Transformer-XL: Attentive Language Models Beyond a

in_channels == out_channels == channels

Swish function

Implements a Transformer Encoder Layer used in BERT/XLM style pre-trained models.

Wrapper class of torch.transpose() for Sequential module.

Wrapper class of torch.view() for Sequential module.