Apply numeric filtering logic with bounds checking. Args: tag_value: The numeric value to test lower_bound: Optional minimum

Apply the configured tagging pipeline to content. Args: content_dict: Content dictionary with data to tag config: Configurat

Assign audio language tags to all entries in a single JSONL file. This function loads a JSONL file and assigns audio language tags to each v

Calculate Word Error Rate (WER) and error statistics for YODAS predictions. Normalizes target and predicted text, then computes WER and detailed

Calculate Word Error Rate (WER) and error statistics for predictions. Normalizes target and predicted text, then computes WER and detailed error

Calculate percentiles from a list of values. Args: values: List of values to calculate percentiles for Returns: List of

Calculate the total length of a transcript in seconds. Args: transcript_start: Start times for transcript segments (format depends o

Check if transcript text exceeds model context length. Validates whether the transcript tokens fit within the model's context window for

Segment machine-generated transcripts aligned with manual transcript timestamps. Processes machine-generated transcript data by aligning it

Clean CORAAL dataset text according to corpus conventions. Applies CORAAL-specific text normalization including dialect word mapping,

Clean and modify subtitle text content using regex patterns. This function removes common subtitle artifacts including: - Speaker names

Cleanup function for distributed YODAS DDP training. Clears GPU memory cache and destroys the distributed process group to properly shut down

Cleanup function for the distributed training

Cleanup function for the distributed training

Collect and merge reservoir sampling results from worker processes. Args: result_queue: Queue containing paths to temporary result f

Discover and combine all language mapping files in a directory. Args: mapping_dir: Directory containing language mapping files

Convert OLMoASR checkpoint to HuggingFace format using subprocess.

Create appropriate dataset loader for the specified evaluation set. Factory method that instantiates the correct dataset loader class based o

Create audio segment from source audio based on timing information. Extracts a time-based segment from a source audio file and saves it

Create a text clip generator function for subtitles. Returns: Function that creates TextClip objects with consistent styling

Discover all JSONL.gz files in the input directory. Args: input_dir: Directory to search for JSONL files Returns: List

Discover transcript files in a directory. Args: input_path: Directory path to search pattern: File pattern to match (default

Display a formatted report of filtering results. This function generates a step-by-step report showing how many items were modified or f

Display percentiles in a formatted table. Args: percentiles: List of percentile values to display

Download OLMoASR checkpoint from HuggingFace hub.

Ensure the checkpoint directory exists.

Extract text from machine-generated subtitle content. Args: machine_content: Raw machine subtitle content (VTT format) Returns:

Extract text from manual subtitle content. Args: manual_content: Raw manual subtitle content (VTT or SRT format) Returns:

Extract timestamp pairs for a list of utterance IDs. Args: utt_id_list: List of utterance IDs Returns: List of (start,

Forward pass through linear layer with dtype casting. Parameters ---------- x : Tensor Input ten

Generate list of file paths for video processing from a segment directory. Args: seg_dir: Directory containing .npy and .srt files

Generate predictions from model logits and decode them to text for YODAS. Takes model output logits and ground truth targets, converts logits to

Generate predictions from model logits and decode them to text. Takes model output logits and ground truth targets, converts logits to prediction

Convert timestamp pairs to token index ranges for YODAS training. Converts floating-point timestamps (in seconds) to Whisper timestamp token

Generate a new audio segment from combined utterances. Args: segment_info: Tuple of (segment_indices, segment_id) output_dir

Generate a comprehensive processing report. Args: files_processed: Number of files processed processing_time: Total processi

Generate a comprehensive summary report. Args: output_dir: Directory to save the report config_path: Path to configuration f

Generate a video with subtitles and audio from numpy array and transcript. Args: npy_file: Path to numpy file containing audio data

Computes error bars for WER values from multiple CSV files in a directory. Args: data_dir (str): Path to the directory containing CS

Extract text content from a machine-generated segment. Args: mach_segment: MachineSegment object containing machine transcript conte

Extract text content from a segment. Args: segment: Segment object containing transcript content Returns: Extracted tex

Identify languages for audio segments using the classification model. Args: model: Pre-trained SpeechBrain language identification m

Atomically increment the counter by 1. Returns: The new value after incrementing Note: This operati

Initialize both HuggingFace and OLMoASR models.

Loads the model (DDP) checkpoint to resume training Args: exp_name: The experiment name. run_id: The run ID rank: The ran

Loads the model (DDP) checkpoint to resume training Args: exp_name: The experiment name. run_id: The run ID rank: The ran

Loads the model (DDP) checkpoint to resume training Args: exp_name: The experiment name. run_id: The run ID rank: The ran

Load configuration from YAML file. Args: config_path: Path to the YAML configuration file Returns: Dictionary containin

Load configuration from YAML file. Args: config_path: Path to the YAML configuration file Returns: Dictionary containin

Load data from a compressed JSONL file. Args: jsonl_file_path: Path to the compressed JSONL file Returns: List of JSON

Load and parse a JSONL file. Args: file_path: Path to the JSONL file to load Returns: List of parsed JSON objects

Load language mapping from a compressed JSON file. Args: mapping_file_path: Path to the compressed JSON file containing language map

Log information about a video with missing language mapping. Args: video_id: ID of the video missing language mapping jsonl_

Log training examples to Weights & Biases table for YODAS training. Creates detailed logging entries for training samples including audio, pr

Log training examples to Weights & Biases table. Creates detailed logging entries for training samples including audio, predictions, targets,

Log training examples to Weights & Biases table. Creates detailed logging entries for training samples including audio, predictions, targets,

Main entry point for tagged data processing pipeline. This function orchestrates the entire processing workflow: 1. Discovers JSONL file

Perform parallel reservoir sampling on JSONL files in a directory. This function implements a distributed reservoir sampling algorithm that:

Merge manual and machine transcript segments. Args: transcript: Dictionary containing transcript content and metadata shard_

Convert numpy audio array to WAV file. Args: npy_file: Path to input numpy file containing audio data wav_file: Path to outp

Read sample dictionaries from a JSONL file. Args: samples_dicts_file: Path to JSONL file containing sample dictionaries Returns

Parse command-line arguments. Returns: Parsed arguments namespace Raises: SystemExit: If argument parsing fails

Parse command-line arguments. Returns: Parsed arguments namespace Raises: SystemExit: If argument parsing fails

Parse subtitle content based on format detection. Args: transcript_string: Raw subtitle content string Returns: Parsed

Partition files into approximately equal chunks for parallel processing. Args: files: List of file paths to partition num_ch

Prepares the training dataset and dataloader for FSDP training. Creates an AudioTextDataset from sample dictionaries and sets up a distributed

Prepares the training dataset and dataloader. Creates an AudioTextDataset from sample dictionaries and sets up a distributed dataloader for t

Prepares the distributed dataloader for YODAS DDP training. Creates a DistributedSampler and DataLoader configured for distributed training.

Prepares the distributed dataloader for FSDP training. Creates a DistributedSampler and DataLoader configured for distributed training. The s

Prepares the distributed dataloader for DDP training. Creates a DistributedSampler and DataLoader configured for distributed training. The sa

Prepare list of input files from various input types. Args: input_path: Single file path, directory path, or list of file paths

Prepares the AdamW optimizer for YODAS DDP training. Creates an AdamW optimizer with the specified hyperparameters for training the DDP-wrapp

Prepares the optimizer for training Prepares the AdamW optimizer for training Args: model: The model to train lr: The learni

Prepares the optimizer for training Prepares the AdamW optimizer for training Args: model: The model to train lr: The learni

Preprocesses audio data for YODAS training. Loads audio using soundfile (supporting various formats), handles multi-channel audio, op

Preprocesses the audio data for the model. Loads the audio file, pads or trims the audio data, and computes the log mel spectrogram.

Preprocesses the audio data for the model. Loads the audio file, pads or trims the audio data, and computes the log mel spectrogram.

Preprocess a JSONL file into segments. Args: json_file: Path to the JSONL file to preprocess shard: Identifier for the shard

Preprocesses YODAS text data with timestamp-aware tokenization. Handles transcript lists with timing information, supporting both timestamp

Preprocesses the text data for the model. Reads transcript data, tokenizes it, and creates input/target sequences with optional timestamps.

Preprocesses the text data for the model. Reads transcript data, tokenizes it, and creates input/target sequences with optional timestamps.

Process a single Arrow file from the YODAS dataset. Args: arrow_file: Path to the Arrow file audio_output_dir: Directory for