FFN (FeedForward Network) :param dim: model dimension (number of features) :param mult: multiply the model dimension by mult

embed_dim: output dimension for each position pos: a list of positions to be encoded: size (M,) out: (M, D)

grid_size: int of the grid height and width return: pos_embed: [grid_size*grid_size, embed_dim] or [1+grid_size*grid_size, embed_dim] (w/

Perform per-sample random masking by per-sample shuffling. Per-sample shuffling is done by argsort random noise. x: [N, L, D]

Self-Attention module :param dim: model dimension (number of features) :param num_heads: number of attention heads :p

We encode full images (i.e., no masking) by linearly projecting image patches, adding position embeddings, then encoding these inputs