Computes and stores the average and current value

hard swish

Randomly mask out one or more patches from an image. @author: uoguelph-mlrg (https://github.com/uoguelph-mlrg/Cutout/blob/master/util/cutout

r"""Implements stochastic gradient descent (optionally with momentum). Nesterov momentum is based on the formula from `On the importance of i

Wikitext-2 dataset. This is a PyTorch dataset for the pre-processed Wikitext-2 dataset, containing about 2M words. This dataset will produce

AdaFisherW Optimizer: An adaptive learning rate optimizer that leverages Fisher Information for parameter updates. The AdaFisherW optimizer extend

Implements Adahessian algorithm. It has been proposed in `ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning`. Arguments:

r"""Implements Adam algorithm. It has been proposed in `Adam: A Method for Stochastic Optimization`_. Arguments: params (iterable):

SE Module Ref: https://github.com/moskomule/senet.pytorch/blob/master/senet/se_module.py

AdaFisher Optimizer: An adaptive learning rate optimizer that leverages Fisher Information for parameter updates. This class AdaFisher optimiz

r"""Implements Adam algorithm. It has been proposed in `Adam: A Method for Stochastic Optimization`_. Arguments: params (iterable):

A basic Focal Transformer layer for one stage. Args: dim (int): Number of input channels. input_resolution (tuple[int]): Input r

The basic unit of MobileNetV3

Computes diagonal elements of activation covariance matrices for different neural network layers.

Computes the diagonal and the full matrix elements of the covariance matrix of activations ('H') for various layer types in a neural network.

Computes the diagonal and the full matrix elements of the gradient covariance matrix ('S') for various layer types in a neural network. This

Computes diagonal elements of gradient covariance matrices for different neural network layers.

r"""Set the learning rate of each parameter group using a cosine annealing schedule, where :math:`\eta_{max}` is set to the initial lr and :ma

r"""Set the learning rate of each parameter group using a cosine annealing schedule, where :math:`\eta_{max}` is set to the initial lr, :math:`T_{

r"""Densenet-BC model class, based on `"Densely Connected Convolutional Networks" <https://arxiv.org/pdf/1608.06993.pdf>`_ Args: grow

r""" Focal Modulation Networks (FocalNets) Args: img_size (int | tuple(int)): Input image size. Default 224 patch_size (int | tup

r""" Focal Modulation Network Block. Args: dim (int): Number of input channels. input_resolution (tuple[int]): Input resulotion.

hard sigmoid

Decays the learning rate of each parameter group by linearly changing small multiplicative factor until the number of epoch reaches a pre-defined

Decays the learning rate of each parameter group by gamma once the number of epoch reaches one of the milestones. Notice that such decay can h

r"""Sets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate f

r""" Image to Patch Embedding Args: img_size (int): Image size. Default: 224. patch_size (int): Patch token size. Default: 4.

Decays the learning rate of each parameter group by gamma every step_size epochs. Notice that such decay can happen simultaneously with other

Inspired by torch.nn.TransformerEncoderLayer and rwightman's timm package.

AdaFisher Optimizer: An adaptive learning rate optimizer that leverages Fisher Information for parameter updates. The AdaFisher optimizer exte

The AdaFisherBackBone class serves as the base class for optimizers that adjust model parameters based on the Fisher Information Matrix to m

Implements Adahessian algorithm. It has been proposed in `ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning`. Arguments:

r"""Implements Adam algorithm. It has been proposed in `Adam: A Method for Stochastic Optimization`_. Arguments: params (iterable):

r"""Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR). The policy cycles the learning rate betwe

Decays the learning rate of each parameter group by gamma every epoch. When last_epoch=-1, sets initial lr as lr. Args: optimizer (Op

Sets the learning rate of each parameter group to the initial lr times a given function. When last_epoch=-1, sets initial lr as lr. Args:

Multiply the learning rate of each parameter group by the factor given in the specified function. When last_epoch=-1, sets initial lr as lr.

Reduce learning rate when a metric has stopped improving. Models often benefit from reducing the learning rate by a factor of 2-10 once learni

r"""Implements stochastic gradient descent (optionally with momentum). Nesterov momentum is based on the formula from `On the importance of i

r"""Implements stochastic gradient descent (optionally with momentum). Nesterov momentum is based on the formula from `On the importance of i

r"""Implements stochastic gradient descent (optionally with momentum). Nesterov momentum is based on the formula from `On the importance of i