Args: dim (int): Number of input channels. num_heads (int): Number of attention heads in each ViT block.

Disable the redundant torch default initialization to accelerate model creation.

Collects the state dict and dump to disk.

Tokenize a list of strings.

Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (int): size of qu

Add speaker and start/end signal on each round.

Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.

Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.

Make causal mask used for bi-directional self-attention.

Make causal mask used for bi-directional self-attention.

Calculate decomposed Relative Positional Embeddings from :paper:`mvitv2`. https://github.com/facebookresearch/mvit/blob/19786631e330df9f3622e

Build masks and position id for left to right model.

Resize tokenizer and embedding. Note: This is the unoptimized version that may make your embedding size not be divisible by 64.

Given a list of sources, each is a conversation list. This transform: 1. Add signal '### ' at the beginning each sentence, with end s

Recursively unwraps a model from potential containers (as used in distributed training). Args: model (`torch.nn.Module`): The model

Recursively unwraps a model from potential containers (as used in distributed training). Args: model (`torch.nn.Module`): The model

Partition into non-overlapping windows with padding if needed. Args: x (tensor): input tokens with [B, H, W, C]. window_size

Window unpartition into original sequences and removing padding. Args: windows (tensor): input tokens with [B * num_windows, window_s