Append a new message.

Args: dim (int): Number of input channels. num_heads (int): Number of attention heads in each ViT block.

Register a new conversation template.

r"""The original timm.models.layers.weight_init.trunc_normal_ can not handle bfloat16 yet, here we first convert the tensor to float32, apply the

Get a conversation template.

Get the prompt for generation.

Args: input_ids (torch.LongTensor): [b, T] pixel_values (torch.FloatTensor): [b, n_images, 3, h, w] im

Args: pil_img (PIL.Image): [H, W, 3] in PIL.Image in RGB Returns: x (np.ndarray): [3, self.image_size, self

Get the prompt for generation.

Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (int): size of qu

Set the system message.

Convert the conversation to gradio history state.

Update the last output. The last message is typically set to be None when constructing the prompt, so we need to update it in-place a

Calculate decomposed Relative Positional Embeddings from :paper:`mvitv2`. https://github.com/facebookresearch/mvit/blob/19786631e330df9f3622e

Args: image_indices (List[int]): [index_0, index_1, ..., index_j] input_ids (torch.LongTensor): [N] Returns

Applies the SFT template to conversation. An example of conversation: conversation = [ { "role":

Preprocesses the inputs for multimodal inference. Args: prepare_list (List[VLChatProcessorOutput]): A list of VLChatProc

Stream the text output from the multimodality model with prompt and image inputs.

Generate a prompt with history for the deepseek application. Args: text (str): The text prompt. image (str): The image promp

Support file path or base64 images. Args: conversations (List[Dict[str, str]]): the conversations with a list of messages. An examp

Function to predict the response based on the user's input and selected model. Parameters: user_text (str): The input text from the user

Args: prompt (str): the formatted prompt; conversations (List[Dict]): conversations with a list of messages;

Convert the conversation to gradio chatbot format.

Partition into non-overlapping windows with padding if needed. Args: x (tensor): input tokens with [B, H, W, C]. window_size

Window unpartition into original sequences and removing padding. Args: windows (tensor): input tokens with [B * num_windows, window_s

Args: prompt (str): the formatted prompt; conversations (List[Dict]): conversations with a list of messages;

Args: img_size: Input image size. patch_size: Patch size. in_chans: Number of image input channels.

Args: img_size (int): Input image size. patch_size (int): Patch size. in_chans (int): Number of input ima

Args: dim (int): Number of input channels. num_heads (int): Number of attention heads. qkv_bias (bool):

Args: kernel_size (Tuple): kernel size of the projection layer. stride (Tuple): stride of the projection layer.