Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (

Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Prints the number of trainable parameters in the model.

input_ids, target_ids, attention_mask: bsz x s2

Rotates half the hidden dims of the input.

this function also save the trainable parameters and specific name parameters

Selects the best resolution from a list of possible resolutions based on the original size. Args: original_size (tuple): The original

Perform uniform spatial sampling on the images and corresponding boxes. Args: images (tensor): images to perform uniform crop. The di

Perform uniform spatial sampling on the images and corresponding boxes. Args: images (tensor): images to perform uniform crop. The di

Add speaker and start/end signal on each round.

extracts samples only pertaining to this worker's batch

extracts samples only pertaining to this worker's batch

In the stage 2: training the decoding-side alignment via minimize the distance between the representation of signal tokens and captio

In the stage 1: training the encoding-side alignment via image/video/audio caption tasks modality: the input modality for each captio

r""" Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*):

r""" Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*):

r""" Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*):

r""" Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*):

r""" Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*):

r""" Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*):

Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.

In the stage 3: instruction-following training via the instruction dataset.

Make causal mask used for bi-directional self-attention.