Linear layer with column parallelism. The linear layer is defined as Y = XA + b. A is parallelized along its second dimension as A = [A_1, ..

A residual block that can optionally change the number of channels. :param channels: the number of input channels. :param emb_channels: t

A sequential module that passes timestep embeddings to the children that support it as an extra input.

The rotary position embeddings from RoFormer_ (Su et. al). A crucial insight from the method is that the query and keys are transformed b

WebDataset with meta information files Extra Format: in webdataset (tar), for each sample there is a '.id'; for each tar file, th

An attention block that allows spatial positions to attend to each other. Originally ported from here, but adapted to the N-d case. https

A single linear layer which does a 1x1 conv

Linear layer with row parallelism. The linear layer is defined as Y = XA + b. A is parallelized along its first dimension and X along its sec

Dataset to concatenate multiple datasets. Purpose: useful to assemble different existing datasets, possibly large-scale datasets as the c

A downsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determini

Dataset wrapper to randomly mapping indices to original order. Will also enlarge the length

Store some hyper-parameters such as image size and patch size. seq_len = pre_len + image_len + post_len

Vision Transformer with support for patch or hybrid CNN input stage

Masked Autoencoder with VisionTransformer backbone

Transformer block for image-like data. First, project the input (aka embedding) and reshape to b, t, d. Then apply standard transform

An upsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determinin

Very simple multi-layer perceptron (also called FFN)

Very simple multi-layer perceptron (also called FFN)

Image to Patch Embedding

Progress Class Class for calculating and displaying download progress

A module which performs QKV attention and splits in a different order.

Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs

Track a series of values and provide access to smoothed values over a window or the global series average.