ADE-bench

"Lord, forgive me, I've been running, running blind in truth."

— Beyonce / "Freedom" / Lemonade

---

Overview

ADE-bench[^1] is a framework for evaluating AI agents on data analyst tasks. Use it to create realistic testing environments with complex evaluation criteria, that can be shared as simple files and tested in production warehouses, while making it as easy as possible to create new tasks.

Watch an explainer video from Coalesce 2025

It does this by running each task in a separate temporary sandbox, and then evaluating the agent's work against the expected results.

Quickstart

1. Clone this repo

git clone https://github.com/dbt-labs/ade-bench.git
cd ade-bench

2. Install Docker Compose and uv

You might already have these, check with:

uv --version
docker compose version

If you don't, install them according to the links above.

3. Create a virtual environment and install the ade CLI

Reminder: cd into the ade-bench directory first.

uv venv
source .venv/bin/activate
uv pip install -e .

Confirm it is installed by running:

ade --help

Learn more about virtual environments, including activation instructions for Windows.

4. Download the shared DuckDB databases

Databases are distributed via GitHub Releases. Download them with the gh CLI:

gh release download databases --repo dbt-labs/ade-bench --pattern "*.duckdb" --dir shared/databases/duckdb

5. Checkpoint: try running a task without any LLMs

ade run simple001 --db duckdb --project-type dbt --agent sage

You should see output like this:

$ ade run simple001 --db duckdb --project-type dbt --agent sage
14:58:07 | system                           | START        | STARTING HARNESS RUN
14:58:07 | system                           | START        | Run ID: 2025-12-05__14-58-07
14:58:07 | system                           | START        | Running with duckdb + dbt. Found 1 tasks of 1 requested tasks.
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Time     ┃ Task                             ┃ Stage        ┃ Message                                                                                              ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 14:58:32 │ simple001                        │ ✓ ✓ ✓  DONE  │ PASS - dbt test results - Pass: 2, Fail: 0, Total: 2                                                 │
│ ──────── │ ──────────────────────────────── │ ──────────── │ ──────────────────────────────────────────────────────────────────────────────────────────────────── │
│ 14:58:32 │ SUMMARY                          │              │ Run 1 of 1 tasks, 1 successful (100.0%)                                                              │
└──────────┴──────────────────────────────────┴──────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────┘
14:58:32 | system                           | FINISH       | Generated HTML dashboard.
14:58:32 | system                           | FINISH       | Harness run completed: 1 of 1 tasks successful

======================================== RESULTS SUMMARY ========================================

+-------------+----------+----------------+---------+----------+------------+------------+--------+----------------+-----------------+----------------+---------+
| Task        | Result   | Failure Type   | Tests   | Passed   | Passed %   | Time (s)   | Cost   | Input Tokens   | Output Tokens   | Cache Tokens   | Turns   |
|-------------+----------+----------------+---------+----------+------------+------------+--------+----------------+-----------------+----------------+---------|
| simple001   | p        |                | 2       | 2        |            | 0          | $0.00  | 0              | 0               | 0              | 0       |
|             |          |                |         |          |            |            |        |                |                 |                |         |
| TOTAL (n=1) | 100%     |                | 2       | 2        | 100%       | 0          | $0.00  | 0              | 0               | 0              | 0       |
+-------------+----------+----------------+---------+----------+------------+------------+--------+----------------+-----------------+----------------+---------+

For more details, run the command below:
ade view

6. Configure one or more LLM API keys

Make a copy of .env.example and rename it to .env, then provide your preferred API keys in the specified locations. See Configuration for a worked example.

7. Run all tasks

ade run all --db duckdb --project-type dbt --agent claude

8. Go beyond

• Use the dbt Fusion engine instead of dbt Core with --project-type dbt-fusion (set up Snowflake first)
• Enable skills, MCP servers, or both with the --plugin-set flag (see Plugin Sets section)
• Contribute additional tasks or datasets

---

An introduction to how ADE-bench works

ADE-bench has three primary components:

1. Tasks
2. Shared databases
3. Shared dbt projects

Each task represents a request that might be made to an agent. Though each task can include multiple evaluation criteria (e.g., it may require multiple models to be updated, or a model to have the correct SQL query and materialization configuration), the task is the primary unit of evaluation in ADE-bench.

When ADE-bench is asked to solve a task, here is what happens:

1. Copy the project into a sandbox. ADE-bench creates a sandbox environment (i.e., a Docker container) for the task. It loads the corresponding project into the container, and creates a sandbox environment for the corresponding database (see "How databases work" below).
2. Take a first snapshot. Once the project is set up, ADE-bench takes a snapshot of all the files in the project. This allows it to log the changes made by additional setup tasks and by the agent.
3. Run task-specific setup script. After taking a snapshot, ADE-bench runs any of the task's additional setup scripts. These might make changes to the project, update the data in the database, or make changes to the project so that it can be run against a different type of databases (see "Sharing projects across databases.")
4. Take another snapshot. ADE-bench takes another snapshot to log what changes made in the step above.
5. Ask the agent to do stuff. The environment is handed to the agent, which then tries to resolve the task.
6. Take a final snapshot. Once the agent declares itself done, ADE-bench takes a third snapshot.
7. Evaluate the result. The changes are evaluated against the tests specified in the task. If all the tests pass, the task passes. Note: ADE-bench includes automatic ways to compare tables to one another. For example, if you want to evaluate a task by seeing if the agent created the correct dim_users table, you can define this table in the task configuration, and the comparison test is generated automatically.
8. Clean up the sandbox. Once the task has recorded its results, ADE-bench deletes the container.

---

Usage

ADE-bench runs with the ade CLI command. Here is an example command for running tasks:

ade run \
  foo001 foo002 \ # The tasks to run. See the task selection list immediately below for more details.
  --db snowflake \ # Which database variant to run
  --project-type dbt \ # Which project variant to run (currently dbt or dbt-fusion)
  --agent sage \ # Which agent to use (e.g., `sage`, `claude`, `codex`)

  --model claude-opus-4-5-20251101 \ # Optional; which specific model to use for different agents.
  --exclude_task_ids foo001 \ # Optional; if you run all tasks, you can exclude tasks with this flag.
  --n-concurrent-trials 4 \ #  Optional; sets trial concurrency limit; defaults to 4.
  --n-attempts 1 \ # Optional; sets times to run each task; defaults to 1.
  --max-episodes 50 \ # Optional; the maximum number of calls to an agent's LM; defaults to 50.
  --seed \ # Optional; flag for creating solution seed CSVs !! DESTRUCTIVE !! RUN WITH CAUTION !! SEE BELOW !!
  --no-diffs \ # Optional; disables taking snapshots of diffs for faster performance.
  --persist \ # Optional; keeps the container alive after the trial is over or is aborted.
  --plugin-set none \ # Optional; plugin set name(s) from experiment_sets/plugin-sets.yaml. Defaults to 'none'.
  --tasks-dir /absolute/path/to/tasks \ # Optional; path to an external tasks directory. Defaults to 'tasks' in the current directory.

See also: complete usage documentation.

Task selection

There are several ways to select tasks to run:

ade run airbnb001  # Run a specific task
ade run f1006.hard  # Run a specific task with a specific variant
ade run airbnb001 airbnb002  # Run multiple tasks
ade run all  # Run all ready tasks
ade run @coalesce  # Run an experiment set
ade run f1+ simple+  # Run wildcard patterns

---

Task Configuration

Before getting into the details, it's useful to see how tasks are configured. This will help define the outline for how ADE-bench works, and the sections below will fill in the details.

Each task folder contains a handful of files:

• task.yaml - The task's configuration. This is an important file, so all the details are below.
• setup.sh – A setup script that runs before the agent is given the task, for modifying files and doing other computer stuff.
• setup/ – (Optional) A directory containing files that the setup script can use. For example, suppose you want to replace a_good_working_file.sql with one_with_a_bug.sql. Put the one with a bug here and use setup.sh to replace it. This lets you modify the project for the specific task, without having to make changes to the shared project.
• solution.sh – A script that solves the task. The sage agent is a agent that just runs this script. See "The sage agent" below for more.
• solutions/ – (Optional) Files that are available to the solution script. This is exactly analogous to the /setup directory for the setup script.
• tests/ - dbt tests that are used to evaluate the trial. For a trial to pass, all the tests in this directory must pass. You can add manual tests, and if you include solution_seeds in the task configuration, tests will get added here automatically when a task is run. Automatically generated tests are appended with the name AUTO_. See "How trials are evaluated" below for more.
• seeds/ - CSVs that are used when evaluating the automatically-generated equality tests.

The task is defined in the task.yaml file:

```yaml

task.yaml

task_id: The name of the task, which should match the name of the task directory. status: An indicator of the task's development status. [ready | dev | open] description: A description of what the task does notes: |- Optional notes about the task, things to add to it, issues with it, etc. prompts: # # Tasks can have multiple prompt variants to see how well agents resolve issues # with different levels of detail. The base prompt is required; additional prompt # variants are optional. Each prompt will run as its own trial in its own sandbox. - key: base prompt: |- The prompt to give to the agent. - key: hard prompt: |- Optional prompt variants. author_name: The name of the task author. author_email: The author's email. difficulty: How hard the task is. category: The category of problem the task is testing. tags: # Additional tags to help with organization - dbt - jinja

An optional script to run after the agent runs, and before the tests are evaluated.

This should be used if

1. the agent might update a file as part of the task

2. and might not run dbt,

3. which should be considered a success,

4. but dbt needs to run to evaluate the tests.

test_setup: |- dbt run --select obt_product_inventory

ADE-bench will automatically do several things with the tables listed here:

1. Create solution seeds (called solution__[table_name]) after the agent runs.

2. Create a dbt test that checks for the existence of this table

3. Create a test to compare the table to corresponding solution seed.

As is shown for with the tables below, you can:

1. Exclude some columns from the equality test

2. Only include certain columns in the equality test

3. Disable the creation of both the existence and equality tests.

4. Specify alternate solution seeds (for tasks with multiple valid answers)

solution_seeds: - table_name: dim_products

• table_name: fct_sales include_columns:

  • id
  • item_name_displayed
  • item_name_actual

• table_name: fct_transactions exclude_columns:

  • bribe_amount_usd

• table_name: generated_income_statement exclude_tests:

  • equality_test # Can also be existence_test

• table_name: sec_filing_report alternates:

  • sec_filing_report_with_fudges

These are the different database and project variants that the task supports

If no migration directory is provided, then no migration script will run.

If one is provided, it will be run against the p