A `Prompt` object that wraps prompts to be compatible with non chat models, which use `openai.Completion.create`.

A `Prompt` object that wraps prompts to be compatible with chat models, which use `openai.ChatCompletion.create`. The format expected by cha

Wrap calls to a completion_fn with a prompt template with applicable keyword args. This will pass many args relevant to OpenAI Completion API

Specification for a base eval.

A "recorder" which only logs certain events to the console. Can be used by passing `--dry-run` when invoking `oaieval`.

StringIO that throws an exception when it's read from

Specification for a CompletionFn.

Evaluation classes generally should override two methods: `eval_sample`: Takes in a test sample and a random number generator and rec

Specification for an eval set.

Specification for an eval.

A recorder which logs events to the specified JSON file. This is the default recorder used by `oaieval`.

A `Prompt` encapsulates everything required to present the `raw_prompt` in different formats, e.g., a normal unadorned format vs. a chat form

A recorder which logs events to Snowflake. Can be used by passing `--no-local-run` when invoking `oaieval`.

The standard events for which recording methods are provided are: - `match`: A match or non match, as specified by the `correct` bool, betwee

This Completion Function uses embeddings to retrieve the top k relevant docs from a dataset to the prompt, then adds them to the context before c