Conv + Norm Layer + Activation w/ optional skip connection

Image to Patch Embedding

Convolutional Position Encoding. Note: This module is similar to the conditional position encoding in CPVT.

Convolutional relative position encoding.

Build Trunk Blocks This ended up being somewhat of a cross between https://github.com/tensorflow/tpu/blob/master/models/official/mnasnet/mna

Factorized attention with convolutional relative position encoding class.

Serial block class. Note: In this implementation, each serial block only contains a conv-attention and a FFN (MLP) module.

Multi-Head Attention

Feature Hook Helper This module helps with the setup and extraction of hooks for extracting features from internal nodes in a model by node

Creating a custom dataset for reading the dataset and loading it into the dataloader to pass it to the neural network for finetuning the

Creating a custom dataset for reading the dataset and loading it into the dataloader to pass it to the neural network for finetuning the

TNT Block

Randomly selects a rectangle region in an image and erases its pixels. 'Random Erasing Data Augmentation' by Zhong et al. See https:/

Iterates through each of `model`'s modules and matches modules using unix pattern matching based on `hook_fn_locs` and registers `hook_fn` to the

r"""Implements AdaBelief algorithm. Modified from Adam in PyTorch Arguments: params (iterable): iterable of parameters to optimize or dic

Implements Adafactor algorithm. This implementation is based on: `Adafactor: Adaptive Learning Rates with Sublinear Memory Cost` (see https://

Implements the AdaHessian algorithm from "ADAHESSIAN: An Adaptive Second OrderOptimizer for Machine Learning" Arguments: params (ite

AugMix Transform Adapted and improved from impl here: https://github.com/google-research/augmix/blob/master/imagenet.py From paper: 'AugMix:

A basic Swin Transformer layer for one stage. Args: dim (int): Number of input channels. input_resolution (tuple[int]): Input re

Inverted residual block w/ CondConv routing

Cosine decay with restarts. This is described in the paper https://arxiv.org/abs/1608.03983. Inspiration from https://github.com/all

DepthwiseSeparable block Used for DS convs in MobileNet-V1 and in the place of IR blocks that have no expansion (factor of 1.0). This is an a

Residual block with expansion convolution followed by pointwise-linear w/ stride Originally introduced in `EfficientNet-EdgeTPU: Creating Accele

GSA: using a key to summarize the information for a group to be efficient.

CNN Feature Map Embedding Extract feature map from CNN, flatten, project to embedding dim.

Inverted residual block w/ optional SE Originally used in MobileNet-V2 - https://arxiv.org/abs/1801.04381v4, this layer is often referred to