This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Disable the redundant torch default initialization to accelerate model creation.

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Rotates half the hidden dims of the input.

Rotates half the hidden dims of the input.

Check whether the text violates OpenAI moderation API.

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, th

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, th

Validate the `rope_scaling` configuration.

Validate the `rope_scaling` configuration.

Setup the optimizer. We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the

@func: evaluate [classify] task @task: image + instruction >>> categroy @metric: accuracy

@func: evaluate [REC] task @task: image + instruction >>> categroy @metric: iou, precision, recall

@func: evaluate [REG] task @task: image + cooridinate >>> categroy @metric: accuracy