Generate the input embeds with both text tokens and visual tokens. Args: input_ids (Tensor): [N, L] position_ids (Ten

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

pixel_values: [1, h*w, c*dim]

Rotates half the hidden dims of the input.

r"""The original timm.models.layers.weight_init.trunc_normal_ can not handle bfloat16 yet, here we first convert the tensor to float32, apply the

Applies Rotary Position Embedding with Multimodal Sections to the query and key tensors (https://qwenlm.github.io/blog/qwen2-vl/). Explanation:

Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (

Resize and pad an image to a target resolution while maintaining aspect ratio. Args: image (PIL.Image.Image): The input image.

Perform per-sample random masking by per-sample shuffling. Per-sample shuffling is done by argsort random noise. x: [N, L, D]

pixel_values: [1, 3, 384, 384]

dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow sca