Downloads and returns the task dataset. Override this method to download the dataset from a custom API. :param data_dir: str

Split text into a list on occurrences of the given separation character `sep_char`. The separation character may be escaped by a backslash to

Instantiate and evaluate a model on a list of tasks. :param lm: obj Language Model :param task_dict: dict[str, Task] Dictiona

Instantiate and evaluate a model on a list of tasks. :param lm: obj Language Model :param task_dict: dict[str, Task] Dictiona

Search upward in the directory tree to a maximum of three layers to find and return the package root (containing the 'tests' folder)

Takes a predicted answer and a gold answer (that are both either a string or a list of strings), and returns exact match and the DROP

Process results from OpenAI API response. :param response: dict OpenAI API Response :param ctxlen: int Length of context (so

Generate table of results.

Query OpenAI API for completion. Retry with back-off until they respond

Take a single document and the LM results and evaluates, returning a dict where keys are the names of submetrics and values are the values of

Create a pruned version of a SCROLLS task dataset containing only inputs that are less than `max_tokens` when tokenized by each tokenizer