Run a shell command and return its output

Prints a summary of statistics for a list of prompts.

Helper to save text content to a file, creating parent dirs.

Counts the number of tokens in a given text using the provided encoder. Args: text: The string to count tokens for. encoder:

Identifies the indices of the top 3 performers in a list of values. Args: values: List of tuples (index, value) to rank Returns

Formats a table cell with medal emoji and border styling if ranked. Args: content: The cell content rank_class: Optional ran

Returns a standard warning about auto-generated files.

Runs a git command and raises exception on error. Args: args: List of arguments for the git command (e.g., ["log", "-1", "--format=%

Attempts to infer the programming language from a filename or path. Args: filename: The filename or path to analyze ext_to_l

Reads a JSON file and returns the parsed content.

Removes characters that are problematic for filenames/paths.

Extracts just the filename from a path, discarding directory information. Args: name: The case name or path Returns: Ju

Initializes and returns the async OpenAI client for OpenRouter.

Finds the latest run directory for a case/model and returns its path and cost. # Helper function for checking existing results. Returns:

Creates a reverse map from extension to language.

Constructs the prompt content using the git history.

Calculates token statistics globally and per language, including quartile distribution. Args: prompt_infos: A list of PromptInfo obj

Clone a GitHub repository into the cached-repos directory. Args: repo_name: A GitHub repository name, either as a full URL (https://

Collects metadata for all prompts in the prompts directory. Args: prompts_dir: Path to the directory containing prompts and their me

Collects result metadata for all benchmark cases and models. Args: results_dir: Path to the directory containing benchmark results

Compute OpenAI cost for GPT-5 using provided pricing. Returns: (cost_openai, input_tokens, output_tokens)

Copies selected prompt and expected output files from temporary directory to the final prompts directory. Args: kept_prefixes: A set

Copies static assets to the docs directory.

Creates a section with links to model-specific benchmark case pages.

Creates JavaScript code for initializing and controlling the chart.

Creates a basic CSS stylesheet for the GitHub Pages site.

Creates an HTML section explaining the benchmark with a git merge conflict example.

Creates the HTML footer. Args: include_chart_js: Whether to include the chart JavaScript code. Returns: HTML string for

Creates the HTML header with basic metadata and CSS link.

Creates a section highlighting key findings from the benchmark.

Creates an HTML table showing success rates by programming language. Args: results_metadata: Dictionary mapping (case_prefix, model)

Creates a summary section describing LoCoDiff and Mentat's role. Returns: HTML string for the summary section

Creates an HTML table showing success rates by prompt size quartiles. Args: results_metadata: Dictionary mapping (case_prefix, model

Creates an HTML section for the token-based chart.

Completely removes a directory and recreates it empty.

Determines the token count ranges for each quartile. Args: prompt_tokens_list: List of token counts for all prompts Returns:

Determines which quartile a token count falls into. Args: token_count: The token count to classify quartile_ranges: List of

Extracts content between the first and the last triple backticks (```). Handles optional language identifiers after the first backticks and s

Filters the list of generated prompt statistics based on prompt token limits. Args: stats_list: The initial list of statistics for a

Finds all unique benchmark case prefixes defined in the benchmark metadata (located in prompts_dir). Falls back to scanning prompts_dir if me

Generates a page displaying the raw model response. Args: case_prefix: The benchmark case prefix model: The model name

Generates a page for a specific benchmark case and model. Args: case_prefix: The benchmark case prefix model: The model name

Generates a page showing all benchmark cases with their results across all models. Args: prompt_metadata: Dictionary of prompt metad

Generates raw case data for the token-based chart. Instead of pre-calculating buckets, provides the raw case data for JavaScript to proc

Generates a page displaying just the expected output content. Args: case_prefix: The benchmark case prefix model: The model

Generates a page for a specific model showing all its benchmark cases. Args: model: The model name prompt_metadata: Dictiona

Generates a page displaying just the prompt content. Args: case_prefix: The benchmark case prefix model: The model name

Generates prompts and expected outputs for eligible files in a repository. Iterates through files matching specified extensions, applies dat

Extracts a set of all unique extensions from the language config.

Scans the prompts directory, reads each prompt, counts tokens, and infers language. Args: prompts_dir: Directory containing the '*_p

Queries the OpenRouter Generation Stats API asynchronously for cost and token information. Args: generation_id: The ID of the genera

Gets the full git log history with patches for a file.

Gets the total lines added and deleted for a file from git history.

Sends a prompt to a specified model via OpenRouter asynchronously. Args: prompt_content: The full content of the prompt to send to t

Get the HEAD commit hash of a repository. Args: repo_path: Path to the cloned repository. Returns: The full commit hash

Checks if a file was modified after the threshold timestamp. Args: rel_path: Relative path of the file within the repository.

Check if a sanitized model directory name corresponds to openai/gpt-5*. We store model directory names sanitized with '/' -> '_', so:

Loads the benchmark configuration including language mappings and model display names. Args: config_path: Path to the configuration

Loads the example git history and expected output from files. Returns: Tuple of (git_history, expected_output)

Loads existing metadata and identifies all existing benchmark case prefixes from the prompts directory. Args: prompts_dir: The direc

Loads language configuration from a YAML file. Args: filepath: Path to the YAML configuration file. Returns: A dictiona

Prints the calculated prompt statistics in a formatted way.

Runs a single benchmark case asynchronously, handling errors and saving metadata. Args: benchmark_case_prefix: The unique identifier

Samples prompts from the filtered list to meet the num_prompts target. Uses targeted sampling to aim for distribution across the token range.

Appends the current generation run's metadata to the list of existing runs and saves the updated list to metadata.json. Args: ex

Save the git log and final file state to files in the benchmark directory

Set up a temporary git repository

Convert various GitHub repository reference formats to a standard 'org/repo' format. Args: repo_name: A GitHub repository name, eith

Update a single metadata.json file. Returns True if updated, False if skipped.

Writes chart data to a JSON file in the specified directory.

Writes the prompt and expected output content to their respective files.

Creates an HTML table showing overall statistics for each model. Args: results_metadata: Dictionary mapping (case_prefix, model) to

Creates a table of contents section for the main page. Returns: HTML string for the table of contents section

Extracts success rate percentage from a cell content string. Args: cell_content: String containing success rate info (e.g. "24.50% (

Calculate Wilson score interval for a binomial proportion. This is used to compute confidence intervals for success rates. Args: