Returns `t5` style ROUGE scores. See the related implementation: https://github.com/google-research/text-to-text-transfer-transformer

Whether this task supports decontamination against model training set.

Taken from the official evaluation script for v1.1 of the SQuAD dataset.

:param model: str :param truncate: bool Truncate input if too long (if False and input is too long, throw error)

:param data_dir: str Stores the path to a local folder containing the `Task`'s data files. Use this to specify the pa

LM wrapper that returns cached results if they exist, and uses the underlying LM if not. :param lm: LM Underlying LM :par

SAT Analog Questions is not publicly available. You must request the data by emailing Peter Turney and then download it to a local di

StoryCloze is not publicly available. You must download the data by following https://cs.rochester.edu/nlp/rocstories/ and pass the f

:param data_dir: str Use this to specify the path to manually downloaded JSON test data. This also needs to include t

BuilderConfig for Hendrycks ETHICS. Args: prefix: *string*, prefix to add to the dataset name for path location. features: *l

:param model: str Anthropic model e.g. claude-instant-v1

:param engine: str TextSynth API engine (e.g. `gptj_6B`) :param truncate: bool Truncate input if too long (if

:param engine: str OpenAI API engine (e.g. davinci) :param truncate: bool Truncate input if too long (if Fal

Initializes a HuggingFace `AutoModel` and `AutoTokenizer` for evaluation. Args: pretrained (str): The HuggingFace

Returns a pre-trained pytorch model from a pre-trained model configuration.

Flattens a dict of lists of validated answers. {"number": ['1', '8'], ...} -> [{"number": ['1'], ...}, {"number": ['8'], ...}]