Append a new message.

Get a conversation template.

Get the prompt for generation.

Args: input_ids (torch.LongTensor): [b, T] images (torch.FloatTensor): [b, max_n_images, 3, height, width]

Register a new conversation template.

r"""The original timm.models.layers.weight_init.trunc_normal_ can not handle bfloat16 yet, here we first convert the tensor to float32, apply the

Convert the conversation to gradio history state.

Tokenize text with <image> tags.

clear CUDA memory cache

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Get the prompt for generation.

Rotates half the hidden dims of the input.

Set the system message.

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, th

Preprocesses the inputs for multimodal inference. Args: sample_list (List[VLChatProcessorOutput]): A list of VLChatProce

Applies the SFT template to conversation. Args: conversations (List[Dict]): A List of messages. sft_format (

play the role of format_messages_v2 and get_images_info in the last version

Applies the SFT template to prompts. Args: prompts (str): the non-sft formatted prompt; sft_format (str, opt

Stream the text output from the multimodality model with prompt and image inputs.

Generate a prompt with history for the deepseek application. Args: text (str): The text prompt. images (list[PIL.Image.Image

Args: conversations (List[Dict[str, str]]): the conversations with a list of messages. An example is : [ {

Function to predict the response based on the user's input and selected model. Parameters: user_text (str): The input text from the user

Args: prompt (str): the formatted prompt; conversations (List[Dict]): conversations with a list of messages;

Reset a new message.

Convert the conversation to gradio chatbot format.

Update the last output. The last message is typically set to be None when constructing the prompt, so we need to update it in-place a

Args: prompt (str): the formatted prompt; conversations (List[Dict]): conversations with a list of messages;

Args: img_size: Input image size. patch_size: Patch size. in_chans: Number of image input channels.

DeepseekV2RMSNorm is equivalent to T5LayerNorm