Generate greedily until a stopping sequence :param requests: list A list of pairs (context, until) context: str

Uses RequestFactory to construct Requests and returns an iterable of Requests which will be sent to the LM. :param doc: T

- context_len allows for a rolling window context, allowing each prediction window to potentially condition on some context :param tok

Compute log-likelihood of generating a continuation from a context. Downstream tasks should attempt to use loglikelihood instead of other

Register a string as contamination to be removed, e.g. a test set This breaks the dirt_string into ngrams to store for future cleaning

Whether the task has a validation set

Whether the task has a test set

Compute full log-likelihood of a string, with no truncation, for perplexity computation - We will use the full max context length of the model

Splits a string into ngram words

Returns a fewshot context string that is made up of a prepended description (if provided), the `num_fewshot` number of examples, and an append

:return: Iterable[obj] A iterable of any object, that doc_to_text can handle

:return: Iterable[obj] A iterable of any object, that doc_to_text can handle

Format refs and preds for sacrebleu corpus calculation. It is very particular

:returns: {str: [metric_score] -> float} A dictionary where keys are the names of submetrics and values are functions

Whether the task has a training set

Takes output from get_rolling_token_windows and makes the context not overlap with the continuation

Returns a pre-trained pytorch model from a pre-trained model configuration.

Returns a pre-trained tokenizer from a pre-trained tokenizer configuration.

Converts `dtype` from `str` to torch.dtype when possible.

inps: a torch tensor of shape [batch, sequence] the size of sequence may vary from call to call returns: a torch tensor of s

Preprocess the dataset into a list of (text, label) tuples.

Given a `doc`, flatten it out so that each JSON blob contains exactly one question and one answer. Logic taken from the reference impl

Whether to include special tokens in encoded text. This should be determined by whether or not the model was trained with special tokens.

:returns: {str: [metric_score] -> float} A dictionary where keys are the names of submetrics and values are functions