Take a single document and the LM results and evaluates, returning a dict where keys are the names of submetrics and values are the values of

Reconstruct and mix transformed outputs back into the original input sequence shape. Key operations: 1. Reshapes and transposes `transfo

Reconstruct and mix transformed outputs back into the original input sequence shape. Key operations: 1. Reshapes and transposes `transfo

Serializes a given function or string. If 'keep_callable' is True, the original callable is returned. Otherwise, attempts to return t

Set or update the configuration for a given key.

Query TextSynth API for completion. Retry with back-off until they respond.

:return: Iterable[obj] A iterable of any object, that doc_to_text can handle

Splits a string into ngram words

:param engine: str OpenAI API engine (e.g. gpt-3.5-turbo-instruct) :param truncate: bool Truncate input if t

Initializes an HF or HF-compatible PreTrainedModel from scratch inside HFLM, using the kwargs passed into self.__init__(). A

Helper method during initialization. Create a tokenizer object corresponding to the correct tokenizer for value of `pretrain

Returns the kwargs needed to apply `accelerate` in `AutoModel.from_pretrained`.

Helper method during initialization. Determines the backend ("causal" (decoder-only) or "seq2seq" (encoder-decoder)) model ty

Creates an dictionary of tasks index with the following metadata, - `type`, that can be either `task`, `python_task`, or `group`.

get logits for the entire sequence :param input_ids: torch.Tensor A torch tensor of shape [batch, sequence_cont]

Lower text and remove punctuation, articles and extra whitespace.

Process logprobs and tokens. :param tokens: list Input tokens (potentially left-truncated) :param outputs: RequestOutput

:returns: {str: [metric_score] -> float} A dictionary where keys are the names of submetrics and values are functions

Wrapper function around the Anthropic completion API client with exponential back-off in case of RateLimitError. params: client: anth

Iterates over FilterEnsembles and applies them to instances

Split dataset into batches of exactly block_size length

Build a set of Instances for a task, and store them in task.instances

Used for byte-level perplexity metrics in rolling loglikelihood

Downstream loglikelihood_rolling perplexity tasks with custom word boundaries should override this!

Downstream tasks with custom word boundaries should override this!

Creates an instance of the LM class using the given argument string and additional config. Parameters: - arg_string: A strin

Divide the elements from *iterable* into *n* parts, maintaining order. >>> group_1, group_2 = divide([1, 2, 3, 4, 5, 6], 2) >>> l

Returns the config as a dictionary.

Instantiate and evaluate a model on a list of tasks. :param lm: obj Language Model :param task_dict: dict[str, Task] Dictiona

Evaluate perplexity on the entire dataset

:return: Iterable[obj] A iterable of any object, that doc_to_text can handle

Search upward in the directory tree to a maximum of three layers to find and return the package root (containing the 'tests' folder)

Gets the git commit hash of your current repo (if it exists). Source: https://github.com/EleutherAI/gpt-neox/blob/b608043be541602170bfcfb8ec9