Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).

MLP as used in Vision Transformer, MLP-Mixer and related networks

Classifier head w/ configurable global pooling and dropout.

Selectable global pooling layer with dynamic input kernel size

2D Image to Patch Embedding

A DataProto is a data structure that aims to provide a standard protocol for data exchange between functions. It contains a batch (TensorDict

Computes and stores the average and current value

MLP using 1x1 convs that keeps spatial dims

r"""Applies a linear transformation to the incoming data: :math:`y = xA^T + b` Wraps torch.nn.Linear to support AMP + torchscript usage by manual

MobileOne building block. This block has a multi-branched architecture at train-time and plain-CNN style architecture at inference time F

A downsample pooling module supporting several maxpool and avgpool modes * 'max' - MaxPool2d w/ kernel_size 3, stride 2, padding 1 * 'max2' -

Convolutional Position Encoding. Note: This module is similar to the conditional position encoding in CPVT.

Convolutional relative position encoding.

Build Trunk Blocks This ended up being somewhat of a cross between https://github.com/tensorflow/tpu/blob/master/models/official/mnasnet/mna

Factorized attention with convolutional relative position encoding class.

LARS for PyTorch Paper: `Large batch training of Convolutional Networks` - https://arxiv.org/pdf/1708.03888.pdf Args: params (i

SE Module as defined in original SE-Nets with a few additions Additions include: * divisor can be specified to keep channels % div == 0 (

Serial block class. Note: In this implementation, each serial block only contains a conv-attention and a FFN (MLP) module.

Multi-Head Attention

Channels-last multi-head attention (B, ..., C)

TNT Block

Conv + Norm Layer + Activation w/ optional skip connection

Feature Hook Helper This module helps with the setup and extraction of hooks for extracting features from internal nodes in a model by node

LayerNorm for channels of '2D' spatial NCHW tensors

https://arxiv.org/abs/2212.00794

We assume the dataset contains a column that contains prompts and other information

Resize and Keep Aspect Ratio

Rotary position embedding w/ concatenatd sin & cos The following impl/resources were referenced for this impl: * https://github.com/lucidrai

Squeeze-and-Excitation w/ specific features for EfficientNet/MobileNet family Args: in_chs (int): input channels to layer rd_rat

SwiGLU NOTE: GluMLP above can implement SwiGLU, but this impl has split fc1 and better matches some other common impl which makes mapping che

Layer config context manager that allows setting all layer config flags at once. If a flag arg is None, it will not change the current value.

r"""Implements AdaBelief algorithm. Modified from Adam in PyTorch Arguments: params (iterable): iterable of parameters to optimize or dic

Implements a pytorch variant of Adan Adan was proposed in Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models[J]

BatchNorm + Activation This module performs BatchNorm + Activation in a manner that will remain backwards compatible with weights trained wit