Masked Autoencoder with VisionTransformer backbone

r""" Swin Transformer A PyTorch impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows` - https://arxiv

GSA: using a key to summarize the information for a group to be efficient.

Track a series of values and provide access to smoothed values over a window or the global series average.

SparseConv module that operates on torch.Tensor

A basic Swin Transformer layer for one stage. Args: dim (int): Number of input channels. input_resolution (tuple[int]): Input re

LSA: self attention within a group

r""" Image to Patch Embedding Args: img_size (int): Image size. Default: 224. patch_size (int): Patch token size. Default: 4.

Image to Patch Embedding

We need to first convert the sparse tensor to dense tensor and then perform DWConv because spconv does not support `groups` option.

Image to Patch Embedding

r""" Swin Transformer Block. Args: dim (int): Number of input channels. input_resolution (tuple[int]): Input resulotion.

r""" Window based multi-head self attention (W-MSA) module with relative position bias. It supports both of shifted and non-shifted window. A

alias Twins-SVT

Use useful results from CPVT. PEG and GAP. Therefore, cls token is no longer required. PEG is used to encode the absolute position on the

r""" Patch Merging Layer. Args: input_resolution (tuple[int]): Resolution of input feature. dim (int): Number of input channels.

SparseConv module that operates on torch.Tensor

SparseConv module that operates on torch.Tensor

SubManifold Conv module that operates on torch.Tensor