Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).

Classifier head w/ configurable global pooling and dropout.

Image to Patch Embedding

MLP as used in Vision Transformer, MLP-Mixer and related networks

r"""Applies a linear transformation to the incoming data: :math:`y = xA^T + b` Wraps torch.nn.Linear to support AMP + torchscript usage by manual

Continuous-time Gaussian diffusion https://arxiv.org/pdf/2107.00630.pdf

Discrete-time Gaussian diffusion https://arxiv.org/abs/2006.11239

Subclass torch's LayerNorm to handle fp16.

Selectable global pooling layer with dynamic input kernel size

Convolutional Position Encoding. Note: This module is similar to the conditional position encoding in CPVT.

Convolutional relative position encoding.

Build Trunk Blocks This ended up being somewhat of a cross between https://github.com/tensorflow/tpu/blob/master/models/official/mnasnet/mna

Factorized attention with convolutional relative position encoding class.

LARS for PyTorch Paper: `Large batch training of Convolutional Networks` - https://arxiv.org/pdf/1708.03888.pdf Args: params (i

Serial block class. Note: In this implementation, each serial block only contains a conv-attention and a FFN (MLP) module.

Multi-Head Attention

TNT Block

MLP using 1x1 convs that keeps spatial dims

Feature Hook Helper This module helps with the setup and extraction of hooks for extracting features from internal nodes in a model by node

1×1 convolution with both weight and bias initialized to zero. 用于 ControlNet / Adapter 的零卷积。

r"""Implements AdaBelief algorithm. Modified from Adam in PyTorch Arguments: params (iterable): iterable of parameters to optimize or dic

DropBlock. See https://arxiv.org/pdf/1810.12890.pdf

Implements a pure pytorch variant of FuseLAMB (NvLamb variant) optimizer from apex.optimizers.FusedLAMB reference: https://github.com/NVIDIA/DeepL

MADGRAD_: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization. .. _MADGRAD: https://arxiv.org/abs/2101.1

Randomly selects a rectangle region in an image and erases its pixels. 'Random Erasing Data Augmentation' by Zhong et al. See https:/

Iterates through each of `model`'s modules and matches modules using unix pattern matching based on `hook_fn_locs` and registers `hook_fn` to the

Implements Adafactor algorithm. This implementation is based on: `Adafactor: Adaptive Learning Rates with Sublinear Memory Cost` (see https://

Implements the AdaHessian algorithm from "ADAHESSIAN: An Adaptive Second OrderOptimizer for Machine Learning" Arguments: params (ite

This is much like `.vision_transformer.Attention` but uses *localised* self attention by accepting an input with an extra "image block" dim