Return the number of items in the dataset.

Return a string representation of the dataset.

Check if the given string is a number

Convert results data to pandas DataFrame for visualization. Args: results_data (dict): evaluation results Returns:

Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.

Exact all forms of numbers from a string with regex.

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, th

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, th

Internal method to handle generation logic using the model. Processes input list and returns inference outputs.

Extract key responses from the model output

Check if image has any transparent pixels

Forward HF model and compute relative probability of safe/unsafe token. Args: queries (List[Tuple[int, str]]): List of prompts.

Generate response using HF model. Args: queries (List[str]): List of prompts. max_new_tokens (int, optional): Maximum

Initialize process group.

Launch Gradio interface for results visualization. Args: results (dict): evaluation results

Load and cache vLLM model.

Make causal mask used for bi-directional self-attention.

Match the last uppercase letter in the string, either in parentheses or standalone. Args: text (str): Input text matc

Apply the RMSNorm normalization to the input tensor. Args: x (torch.Tensor): The input tensor. Returns:

Apply the RMSNorm normalization to the input tensor. Args: x (torch.Tensor): The input tensor. Returns:

Normalize the str to lower case and make them float numbers if possible.

Preprocess an image or batch of images. Copy of the `preprocess` method from `CLIPImageProcessor`. Args: images (`ImageI

Validate the `rope_scaling` configuration.

Generate response using vLLM model. Args: queries (List[str]): List of prompts. max_new_tokens (int, optional): Maxim

Add KL divergence regularization on scalar rewards.

Add KL divergence regularization on scalar rewards.

Add KL divergence regularization on scalar rewards.

Add KL divergence regularization on scalar rewards.

Add KL divergence regularization on scalar rewards.

Process responses with both letter matching and open-ended analysis

Process responses with both letter matching and open-ended analysis in Latex format.

Compare model answer list with ground truth Args: data_1: Model answer list data_2: Ground truth (str representing fl