The actual logic for judging belief using LLM judge. Args: subject_inputs: List of SubjectInputForLLMJudge objects. completi

Parse numbered list from LLM completion text that follows a specific format. The text should contain sections marked with `start_marker` and

Finds all locations of "unknown" in the prompt. The prompt format is generally like this: token<tab>unknown unknown_tokens is a hack for

Returns (exp_name_without_seed, seed). Defaults to seed=64.

Process activations for a batch, handling both pre-computed and subject activations. Context: pre-computed activations can be part of the da

Sends messages using the ChatConversation. Handles interventions and updates activations after each token.

Cache a batch of completions.

Convert batched activations to list of individual activations

Add two Usage objects together, combining their token counts.

AND over all neurons in l1 and l2 in O(N). This is complex since the neurons may have `None` tokens or `None` polarity, which are cat

Copy the DB file (and configs dir) from *old_db_dir* → *new_db_dir*. Returns the **path** to the copied DB file.

Extract token counts from OpenAI-style usage payloads.

Create a provider client with an optional explicit API key.

Recursively applies a filter to a list of neurons. If neurons is None, that means all neurons are in the set.

Get the generation prompt for a general question type

Get the generation prompt for a given attribute class

Get the generation prompt for a given attribute class

Creates appropriate iterator for dataset type.

Here's how we get run metadata: - All activations are returned for each (layer, neuron, token) - The current filter may have

Get the validation prompt for a given attribute class

Open an async connection (read-only if requested).

OR over all neurons in l1 and l2 in O(N). This is complex since the neurons may have `None` tokens, which is a catch-all. TO

Parse the LLM's generation response into SystemPrompt objects

Parse the LLM's generation response into AttributePrompt objects

Parse the LLM's validation response

Get the API key for a provider from explicit keys or environment.

Rotate to the next API key for the current provider. If all keys for the current provider are exhausted, move to the next provider.

Synchronously create the table and indices if they don't exist. Called from __init__ for non-read-only instances.

Add a list of tensors for the given layer and module

Issue ``ALTER TABLE`` statements for any *missing* columns. The set of desired columns (and their SQLite types) is taken from the subclass' `

Aggregate eval results saved during training and combine with standalone evaluation results. During distributed training, each rank saves ev

Build SubjectInput objects with eval and baseline results bundled together. For each key, queries both eval_db and baseline_db, computes faithful

Cluster a list of neurons. These should already be filtered to interesting neurons. Neurons will be automatically deduplicated.

:param samples: List of tuples containing a prompt and a completion :param shift_labels: Okay hear me out, this is stupid. By default