Conv2d + BN layer with folding capability to speed up inference Can be merged with Conv() function with additional arguments

Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).

Vision Transformer A PyTorch impl of : `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale` - https://arxiv.org/

Modified to support layer fusion

Efficient RADIO

Execute the function on rank 0 first, followed by all other ranks. Useful when caches may need to be populated in a distributed environment.

Standard bottleneck.

Standard bottleneck.

Faster Implementation of CSP Bottleneck with 2 convolutions.

Convolutional block, used in first couple of stages Experimented with plan resnet-18 like modules, they are the best in terms of throughput

Down-sampling block Pixel Unshuffle is used for down-sampling, works great accuracy - wise but takes 10% more TRT time

E-RADIO Layer

The block is used to output dense features from all stages Otherwise, by default, only the last stage features are returned with E-RADIO

Multi-Layer Perceptron (MLP) block

MultiResolutionAttention (MRA) module The idea is to use multiple attention blocks with different resolution Feature maps are downsampled

Patch embedding block Used to convert image into an initial set of feature maps with lower resolution

Patch Merging Layer. Downsample features by merging 2x2 neighboring patches.

Patch Splitting Layer - Inverse of PatchMerging. Upsample features by splitting each patch into 2x2 neighboring patches.

2D positional embedding from Swin Transformer v2 Added functionality to store the positional embedding in the model and not recompute it ever

A model that learns to predict the position of a patch token in the output of a RADIO model.

Decoder for RADIO1D that reconstructs the original sequence length and embedding dimension. Takes compressed global tokens from the encoder and r

Pretrained Hugging Face configuration for RADIO models.

Pretrained Hugging Face model for RADIO. This class inherits from PreTrainedModel, which provides HuggingFace's functionality for loading and

https://arxiv.org/pdf/2502.05795v1

Implementation of xPos based on RotaryEmbedding from GPT-NeoX. This implementation is designed to operate on queries and keys that are compatible

Just a batchnorm.

Faster Implementation of CSP Bottleneck with 2 convolutions.

2D image to patch embedding: (B,C,H,W) -> (B,N,D) Args: img_size: Image size. patch_size: Patch token size. in_chans

Vision Transformer A PyTorch impl of : `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale` - https://arxiv.org/