r""" MetaFormer A PyTorch impl of : `MetaFormer Baselines for Vision` - https://arxiv.org/abs/2210.13452 Args: in_chan

Scale vector by element multiplications.

Vanilla self-attention from Transformer: https://arxiv.org/abs/1706.03762. Modified from timm.

Implementation of one MetaFormer block.

Absolute pos embedding, learned.

This is a more standard version of the position embedding, very similar to the one used by the Attention is all you need paper, generalized t

Downsampling implemented by a layer of convolution.

r""" General LayerNorm for different situations. Args: affine_shape (int, list or tuple): The shape of affine weight and bias.

MLP as used in MetaFormer mldanbooru, eg Transformer, MLP-Mixer, PoolFormer, MetaFormer baslines and related networks. Mostly copied from timm.

MLP classification head

Implementation of pooling for PoolFormer: https://arxiv.org/abs/2111.11418 Modfiled for [B, H, W, C] input

r""" Inverted separable convolution from MobileNetV2: https://arxiv.org/abs/1801.04381.

Squared ReLU: https://arxiv.org/abs/2109.08668

StarReLU: s * relu(x) ** 2 + b