Loader for IndexedDataset

The rotary position embeddings from RoFormer_ (Su et. al). A crucial insight from the method is that the query and keys are transformed b

Uses aiter.flash_attn_func when there is NO padding. Falls back to MHA-style expansion if num_key_value_heads < num_attention_heads (GQA).

Implement the scaled dot product attention with softmax. Arguments --------- softmax_scale: The temperature to use for the softmax att

Implement the scaled dot product attention with softmax. Arguments --------- softmax_scale: The temperature to use for the softmax att

Head for sentence-level classification tasks.

Head for sentence-level classification tasks.

Head for sentence-level classification tasks.

MLP using 1x1 convs that keeps spatial dims

Implement the scaled dot product attention with softmax. Arguments --------- softmax_scale: The temperature to use for the softmax att

Controls collating in the DataLoader The CollateMixin classes instantiate a dataloader by separating collate arguments with the rest of the datal

Counter the number of FLOPs used by the model

An abstract class to handle weights initialization and a simple interface for dowloading and loading pretrained models.

MLP as used in gMLP

MLP w/ GLU style gating See: https://arxiv.org/abs/1612.08083, https://arxiv.org/abs/2002.05202

Inputs/outputs are BNHD (batch, seq, heads, dim), like your harness. Forward: O, L via tk_kernel_fwd.dispatch_fwd Backward: dQ,dK,dV vi

Implement the scaled dot product attention with softmax. Arguments --------- softmax_scale: The temperature to use for the softmax att

Uses HipKittensFlashAttnFn when there is NO padding. Falls back to MHA-style expansion if num_key_value_heads < num_attention_heads (GQA).

Inputs/outputs are BNHD (batch, seq, heads, dim), like your harness. Forward: O, L via tk_kernel_fwd.dispatch_fwd Backward: dQ,dK,dV vi

Implement the scaled dot product attention with softmax. Arguments --------- softmax_scale: The temperature to use for the softmax att

self.collate_fn(resolution, img_size) produces a collate function that resizes inputs to size img_size/resolution

Iris instance wrapper for Python

Iris Tensor wrapper class

Generate confusion matrix every epoch and send it to wandb. Expects validation step to return predictions and targets.

Generate f1, precision, recall heatmap every epoch and send it to wandb. Expects validation step to return predictions and targets.

Logs a validation batch and their predictions to wandb. Example adapted from: https://wandb.ai/wandb/wandb-lightning/reports/Image-Classif

Monitor the loss scale for AMP (fp16).

Multi-head self-attention and cross-attention

MLP as used in Vision Transformer, MLP-Mixer and related networks

MLP as used in Vision Transformer, MLP-Mixer and related networks

Monitor the scales of weights and gradients.

Keep track of how many tokens we've seen.

Log the number of parameters of the model

r""" Perplexity measures how well a language model predicts a text sample. It's calculated as the average number of bits per word a model need

r"""Inject some information about the relative or absolute position of the tokens in the sequence. The positional encodings have the same dime

Implement the scaled dot product attention with softmax. Arguments --------- softmax_scale: The temperature to use for the softmax att

self.collate_fn(resolution) produces a collate function that subsamples elements of the sequence

Monitor the speed of each step and each epoch.

Timing result structure

Timing result structure

Wrap timm.scheduler.CosineLRScheduler so we can call scheduler.step() without passing in epoch. It supports resuming as well.

Wrap timm.data.Mixup that avoids the assert that batch size must be even.

Upload checkpoints to wandb as an artifact, at the end of run.

Upload all code files to wandb as an artifact, at the beginning of the run.

Upload all code files to wandb as an artifact, at the beginning of the run.

Make wandb watch model at the beginning of the run.