Load the original dataset

Make causal mask used for bi-directional self-attention.

Identifies every occurrence of "System:" in `input_ids` (tokenized form), then masks (sets to IGNORE_INDEX) from the first token of "System:"

Identifies every occurrence of "System:" in `input_ids` (tokenized form), then masks (sets to IGNORE_INDEX) from the first token of "System:"

Identifies every occurrence of "User:" in `input_ids`, then masks (sets to IGNORE_INDEX) from that token to the next "<end_of_utterance>"

Identifies every occurrence of "User:" in `input_ids`, then masks (sets to IGNORE_INDEX) from that token to the next "<end_of_utterance>"

Merges attn_head_size dim and num_attn_heads dim into hidden_size

When an image URL appears more than once, when the HTML is rendered, the same image is displayed. The trick is to add a `_` at the end of the

Prepare model, optimizer and dataloader if necessary. Register the scheduler for checkpointing.

It could be interesting to remake a tree after the simplifications since it can now merge some text nodes that couldn't be merged befo

Function used to remove empty leaves iteratively, so it also ends up also removing nodes that are higher up in the tree.

Post-processes a conversation to handle missing or expanded "image"/"video" tokens based on the loaded frames. If there's no explici

Post-processes a conversation to handle missing or expanded "image"/"video" tokens based on the loaded frames. If there's no explici

Prepare variables for trainings.

This function is mainly used in Localized Narratives where a paragraph can contain multiple relevant captions to a single image. We split the para

Strips all nodes with tags NOT in INTERESTING_TAGS_SET and has counterintuitively nothing to do with the STRIP_TAGS list

Warm up the model with a test prompt

derive and return `torch_dtype` to be used in `from_pretrained` from either Deepspeed config or if Deepspeed isn't used than accelerator stat

Return True if the dataset entry should always be sampled at 100%. We check for keywords like "gpt", "visualwebinstruct", or "vision_flan".

Monkey-patch HF's `_get_unpad_data` with `_my_get_unpad_data_varlen`. This modifies the varlen logic for "flash_attention_2".

Detect if there's a previous checkpoint to resume from. Return True if we found a checkpoint. Otherwise, we start fresh.

Compute the overall target fraction (a float in [0,1]) for the dataset entry based on its modality and video duration if applicable.

1) Load a YAML describing multiple sub-datasets. 2) Create a list of SupervisedDataset objects. 3) If data_args.packed => use PackedConca

Build, shard and save

Args_ : Media Node Returns : clip_score_max_per_img: Max CLIP score per Image clip_nbr_per_img: Number of CLIP scores

Heuristics to decide whether to restart this opt_step-XXX checkpoint evaluation or not

Args_ : Media Node Returns : img_has_good_quality: Boolean indictating there is an image with good quality (defined by its height