Helper for more ergonomic access to a single CompletionFn.

Evaluate all provided samples in parallel.

Returns the number of tokens used by a list of messages.

Render a chat prompt as a text prompt. User and assistant messages are separated by newlines and prefixed with "User: " and "Assistant: ", re

Returns n_ctx for a given API model name. Model list last updated 2023-11-16.

ARGS ==== `answers`: A sequence of strings, each of which is an answer choice. `correct_idx`: The integer index of the correct answer

Format a template string with only necessary kwargs.

Lower text and remove punctuation, articles and extra whitespace.

Return the actual data to be passed as the `prompt` field to your model. See the above types to see what each API call is able to han

Load registry from a list of paths. Each path or yaml specifies a dictionary of name -> spec.

Create a CompletionFn. The name can be one of the following formats: 1. openai-model-id (e.g. "gpt-3.5-turbo") 2. completion-

Can read/write to files on gcs/local with or without gzipping. If file is stored on gcs, streams with blobfile. Otherwise use vanilla python open.

Shuffle `samples` and pair each sample with its index.

Returns the scores

Format a template string with kwargs.

For each path in the input, iterate over the jsonl files in that path. Look in subdirectories recursively. Use an iterator to conserve m

Lower text and remove punctuation, extra whitespace for Chinese characters.

Clean Numbers in the given string >>> _clean_numbers(None, "Hello 123") 'Hello 123' >>> _clean_numbers(None, "Hello 1234") 'Hell