Config for `Evaluator` objects that compute metrics

Dataset, its sampling rate, and its token weight

Implements the beam search algorithm for decoding the most likely sequences. :param end_index: The index of the "stop" or "end" token in the

An error with a configuration file.

How to weight a message

The expected shape and dtype of a tensor

Boxes to draw on an image

Infinitely iterates over a mixture of datasets

Config for a vision transformer

Configuration for a multi-layer transformer

Vision ViT and the Image/Language Connector

Evaluation dataset for video object tracking. Contains 3.7k examples across 5 source datasets with ground-truth masks for computing segmentat

Training dataset for video object tracking with pre-computed instruction data. This is the primary class used for training. It loads from the ins

Video preprocessor, converts videos into crops and tokens

Applies prompt templates and adds system prompts to construct text inputs/output Includes methods for formatting points, video points, and a

Configuration for a torch `DataLoader`

MeViS: https://github.com/henghuiding/MeViS

Molmo2 preprocessor configuration

r""" This is the configuration class to store the configuration of a [`Molmo2TextModel`]. It is used to instantiate a `Molmo2TextModel` accord

r""" This is the configuration class to store the configuration of a [`Molmo2VisionTransformer`]. It is used to instantiate a `Molmo2VisionTra

MDPA for the image ViT

Dataset wrapper that supports padding and control the random seed based on the epoch

Configures FSDP

r""" This is the configuration class to store the configuration of Molmo2Adapter. With Molmo2VitConfig, It is used to instantiate an Molmo2Vis

Config for preprocessing and tokenizing image(s)

An error from incorrect environment variables.

Indicates a key can have a variable shape after collation

YouCook2 dataset for video clip captioning and action localization.

ActivityNet Video dataset (Captioning / ActivityNetQA)

CharadesSTA Video dataset

Ego4d NLQ/MQ subsets **Please use Ego4dCachedClips for faster training as it uses pre-extracted clips.** NLQ: the query is expressed in text

Group of datasets with a fixed sampling rate

A :class:`FinalSequenceScorer` which scores the sequences by the average log probability of the tokens in the sequence. It optionally include

Molmo2 model configuration

Loads allenai/Molmo2-VideoPoint from HuggingFace. Each example (msg) has: subset, example_id, label, answer, count, points (sorted), timestam

A :class:`~torch.distributed.checkpoint.StorageReader` based on :class:`~torch.distributed.checkpoint.FileSystemReader` that can read data di

Decides how to sample timestamps from a video based on duration

A `Sampler` which redistributes the probability mass function for nodes among the top choices with a cumulative probability of at least `p`,

AI2D dataset builder, this builder adds the labelled boxes as needed to the AI2D images using both transparent and opaque boxes

Cache for attention biases and other things that would normally be stored as buffers. We avoid using buffers because we've run into various i

Annotations from Tracking Any Object Amodally (TAO-Amodal) dataset: https://github.com/WesleyHsieh0806/TAO-Amodal BURST uses the same videos

COIN dataset

CountQa dataset from PaliGemi, it is built by merging the CountBench image/count pairs with the natural language questions from the PaliGemma pape

Configuration for an offline dataset evaluation, it could be for loss or generation

Rotary embedding module that operates on an individual [batch, seq_len] array This is used for rotating patch key/queries in MolmoPoint

Light-weight checkpoint that stores the state of a `IterableDatasetMixture`

LV-VIS: Large Vocabulary Video Instance Segmentation.

Configuration for a loss evaluation

Converts list of examples from our datasets into a tensor batch

Molmo model configuration

PerceptionTest dataset for video understanding evaluation.

ReVOS: Reasoning Video Object Segmentation.

A :class:`~torch.distributed.checkpoint.StorageWriter` that can write directly to both cloud and local storage.

Configures throughput monitoring

Data for visual input that has been converted into tokens

A `Sampler` which redistributes the probability mass function for nodes among the top `k` choices, then samples from that subset after re-nor

Configuration for a training run

ViCaS: Video Camouflaged Animal Segmentation.

Frames from a video and frame metadata

Configures wandb logging

Logs progress to Ai2 beaker's server

ChartQA dataset from HuggingFace M4 project. Can be weighted to balanced human/synthetic data

CinePile Video QA dataset — multi-choice QA on movie clips. Annotations auto-downloaded from HuggingFace. Videos (YouTube movie clips) requir

Countix — repetition counting QA on Kinetics-700 videos. Annotations auto-downloaded from HuggingFace. Videos (Kinetics-700) require manual d

Epic Kitchens 100 dataset for short video clip captioning. Epic Kitchens is a large-scale dataset of egocentric videos in kitchen environments

Configuration for an offline dataset evaluation, it could be for loss or generation

Preprocesses examples dictionaries as returned by our data loaders Includes loading the multi-modal data and formatting the text for the LLM