Defines the operation to perform on a list of the `inst.resps` properties of `Instance` objects. Should return the list of (filtered)

r""" A streamlined generate() method overriding the transformers.GenerationMixin.generate() method. This method uses the same logits

Gets the reordered array Returns: List[Any]: The reordered array

Method for padding a list of tensors given the maximum tensor length in the batch. Used for batching inputs and continuations in seq2seq

:return: Iterable[obj] A iterable of any object, that doc_to_text can handle

:return: Iterable[obj] A iterable of any object, that doc_to_text can handle

Generates and yields batches from the reordered array. Parameters: - n (int): The size of each batch. Defaults to 1.

Restores the original order of elements from the reordered list. Parameters: - newarr (List): The reordered array.

Whether the task has a validation set

Compute max metric between prediction and each ground truth.

Normalization used in official SQuAD evaluation script.

Draw the first `n` samples in order from the specified split. Used for tasks with "canonical" ordered fewshot examples, such as MMLU

Format refs and preds for sacrebleu corpus calculation. It is very particular

Use Pytorch Benchmark on the backward pass of an arbitrary function.

Use Pytorch Benchmark on the forward pass of an arbitrary function.

Create a filtering pipeline.

Restores the original order of a new array based on the old array's order Args: newarr (List[Any]): The array to be restored

Groups elements of an iterable based on a provided function. Parameters: - arr (Iterable): The iterable to be grouped.

Whether the task has a test set

Query OpenAI API for completion. Retry with back-off until they respond

Reconstruct and mix transformed outputs back into the original input sequence shape. Key operations: 1. Reshapes and transposes `transfo

Reconstruct and mix transformed outputs back into the original input sequence shape. Key operations: 1. Reshapes and transposes `transfo

:param inps: torch.Tensor A torch tensor of shape [batch, (sequence_ctx + sequence_cont)] or of shape [batch, sequenc

Reorders the elements in the array based on the sorting function. Parameters: - arr (Union[List, Tuple[Tuple[int, Any], ...]

Use Pytorch Benchmark on the forward+backward pass of an arbitrary function.

Ignore capitalization and determiners.

Uses RequestFactory to construct Requests and returns an iterable of Requests which will be sent to the LM. :param doc: T

Downloads and returns the task dataset. Override this method to download the dataset from a custom API. :param data_dir: str

Extended JSON dump that can handle pydantic models.

Returns a fewshot context string that is made up of a prepended description (if provided), the `num_fewshot` number of examples, and an append

Flatten a multi-level nested collection of dictionaries and lists into a flat dictionary. The function traverses nested dictionaries and

Divides an iterable into chunks of specified size or based on a given function. Useful for batching Parameters: - it

Converts `dtype` from `str` to torch.dtype when possible. Does not use an instantiated HF AutoConfig

Whether the task has a training set

Generate table of results.