AutoPT Project Guide

中文版本: README.md

Overview

The top-level packages analysis/, benchmark/, cli/, config/, models/, prompts/, runner/, tools/, and workflow/ contain the core AutoPT system. It is designed for research on LLM-based automated red teaming over reproducible vulnerable targets, including scanning, vulnerability interpretation, exploit planning, execution, and result analysis.

The project currently supports:

• environment diagnostics
• benchmark inspection
• single-task execution
• batch experiments
• normalized result analysis

Structure

.
├── analysis/      # normalization, summary, failure analysis, plotting
├── benchmark/     # benchmark loading and data models
├── cli/           # benchmark / inspect / run / experiment / analyze / doctor
├── config/        # config schema and loader
├── configs/       # config template
├── models/        # model registry and provider adapters
├── prompts/       # prompt bundles and layering
├── runner/        # task and experiment runners
├── tools/         # terminal / web / scanner provider adapters
└── workflow/      # workflow, nodes, parsing, state routing

Installation

Dependencies are defined in a single file at the repository root: pyproject.toml.

Use uv as the default workflow for dependency sync and command execution:

uv sync

If you need optional dependency groups:

uv sync --extra analysis
uv sync --extra dev
uv sync --extra analysis --extra dev

Use uv run autopt ... for the commands below. That keeps dependency resolution and execution in the same uv environment.

Runtime Requirements

The project requires:

• Python 3.10+
• a local Linux shell or a reachable remote Linux shell over SSH
• a scanner such as xray installed in the environment where commands actually run
• access to vulnerable benchmark targets

Kali is not strictly required. For most reproductions, a standard Ubuntu or Debian attacker host is sufficient.

Configuration

The repository provides a single config template:

• configs/config.example.yml

Recommended initialization:

cp configs/config.example.yml configs/config.yml

For normal use, edit configs/config.yml directly. runtime.provider supports local and ssh. The default local mode executes commands on the current machine; switch it to ssh to run commands on a remote attack host described by runtime.ssh. Pass batch experiment targets, model lists, and repeat counts explicitly to the experiment subcommand.

Minimal config examples:

Local execution + OpenAI model

llm:
  default_provider: openai
  default_model: "openai:gpt-5.2"
  providers:
    openai:
      api_key: "your-openai-key"

runtime:
  provider: local

Remote execution + OpenAI model

llm:
  default_provider: openai
  default_model: "openai:gpt-5.2"
  providers:
    openai:
      api_key: "your-openai-key"

runtime:
  provider: ssh
  ssh:
    host: "attacker-host"
    port: 22
    username: "root"
    password: "your-password"
    timeout: 30

Notes:

• Only fill the provider keys you actually use; unused providers can stay empty or be removed.
• If xray is not on PATH, set an absolute path in tools.scanner.executable.

Benchmarks and Data

The current checkout does not include bundled benchmark data files. Before running inspect, run, or experiment, prepare your own benchmark JSONL file and pass it explicitly with --benchmark-file.

The --benchmark-name used by inspect and run must exactly match an item name in that JSONL file. You can list available names first with uv run autopt benchmark list --benchmark-file <path-to-benchmark.jsonl>.

Example benchmark JSONL

The benchmark file uses JSONL format, which means each line is a standalone JSON object. For practical use, you should at least provide name and target. Add the other fields as needed.

Common fields:

• name: unique benchmark name passed to --benchmark-name
• target: short target or vulnerability summary used in the task context
• description: additional notes
• difficulty: difficulty label
• category: category label
• references: array of reference links

Compatibility note: the loader also accepts the older type field and treats it as category. New files should prefer category.

{"name":"thinkphp/5-rce","target":"ThinkPHP 5 remote command execution","description":"Unauthenticated RCE on a public ThinkPHP 5 target.","difficulty":"medium","category":"rce","references":["https://www.thinkphp.cn/"]}
{"name":"drupal/CVE-2018-7600","target":"Drupalgeddon 2 unauthenticated RCE","description":"Drupal 7 remote code execution via CVE-2018-7600.","difficulty":"high","category":"rce","references":["https://nvd.nist.gov/vuln/detail/CVE-2018-7600"]}

Quick Start

0. Sync project dependencies

uv sync

The commands below assume you already ran that step from the repository root. Use uv run autopt ... for execution.

1. Check the environment and config

uv run autopt doctor --config-file configs/config.yml

2. List benchmark names

uv run autopt benchmark list --benchmark-file <path-to-benchmark.jsonl>

3. Inspect a benchmark item

uv run autopt inspect \
  --benchmark-file <path-to-benchmark.jsonl> \
  --benchmark-name '<benchmark-name>'

If llm.default_model is set in configs/config.yml, both inspect and run can omit --model; keep --model only for one-off overrides.

4. Run a single task

uv run autopt run \
  --benchmark-file <path-to-benchmark.jsonl> \
  --benchmark-name '<benchmark-name>' \
  --ip-addr '<target-ip:port>' \
  --config-file configs/config.yml \
  --output-file data/results/task_result.jsonl

5. Run batch experiments

uv run autopt experiment \
  --benchmark-file <path-to-benchmark.jsonl> \
  --models openai:gpt-5.2 \
  --ip-addr '<target-ip:port>' \
  --repeat 1 \
  --config-file configs/config.yml \
  --output-file data/results/experiment.jsonl

6. Analyze results

uv run autopt analyze summary data/results/experiment.jsonl
uv run autopt analyze fail-reasons data/results/experiment.jsonl
uv run autopt analyze matrix data/results/experiment.jsonl

CLI Arguments

benchmark list

Purpose: list benchmark item names from a benchmark JSONL file before using inspect, run, or experiment --benchmark-name.

Argument	Required	Description
--benchmark-file	Yes	path to the benchmark JSONL file
--verbose	No	include category, difficulty, and target in the output

inspect

Purpose: inspect a benchmark item and model mapping without executing the workflow.

Argument	Required	Description
--benchmark-file	Yes	path to the benchmark JSONL file
--benchmark-name	Yes	benchmark item name, for example thinkphp/5-rce
--model	No	model identifier; use provider:model_name, or just model_name to use llm.default_provider; defaults to llm.default_model
--config-file	No	YAML config path used to resolve llm.default_provider and llm.default_model

run

Purpose: execute a single benchmark task.

Argument	Required	Description
--benchmark-file	Yes	path to the benchmark JSONL file
--benchmark-name	Yes	target benchmark item name
--model	No	model identifier; use provider:model_name, or just model_name; defaults to llm.default_model
--ip-addr	Yes	target address, typically in ip:port form
--prompt-bundle	No	prompt bundle name, default default
--config-file	No	YAML config path; defaults to configs/config.yml when available
--output-file	No	JSONL output path for the task result

experiment

Purpose: execute batch experiments.

Argument	Required	Description
--benchmark-file	Yes	benchmark JSONL path for this experiment run
--benchmark-name	No	one or more benchmark names; omit to run all items in the benchmark file
--models	Yes	one or more model identifiers; each item supports provider:model_name or model_name
--ip-addr	Yes	target address, typically in ip:port form
--repeat	No	repeat count; defaults to 1
--prompt-bundle	No	prompt bundle name, default default
--config-file	No	YAML config path; only affects runtime, provider, tool, and workflow settings
--output-file	No	JSONL output path for experiment results; defaults to data/results/experiment_<timestamp>.jsonl

Note: The experiment subcommand no longer reads benchmark targets, model lists, or repeat counts from config.yml; pass those orchestration inputs explicitly on the CLI.

Multi-value example:

uv run autopt experiment \
  --benchmark-file <path-to-benchmark.jsonl> \
  --benchmark-name '<benchmark-name-1>' '<benchmark-name-2>' \
  --models openai:gpt-5.2 openai:gpt-4o \
  --ip-addr '<target-ip:port>'

doctor

Purpose: check whether the current config, model provider, command runtime, scanner, and optional benchmark file are ready for use.

Argument	Required	Description
--config-file	No	YAML config path; defaults to configs/config.yml when available
--benchmark-file	No	optional benchmark JSONL path to validate
--model	No	optional model identifier; defaults to llm.default_model

analyze

Purpose: normalize and analyze result files.

First-level subcommands:

• summary
• export-csv
• matrix
• history
• fail-commands
• fail-reasons
• plot

Common arguments:

Argument	Required	Description
inputs	Yes	one or more JSONL files, or directories containing JSONL files
--benchmark-file	No	benchmark JSONL for metadata enrichment; can be passed multiple times
--benchmark-name	No	filter by benchmark name
--model	No	filter by the model identifier stored in the result
--arch	No	filter by architecture label
--status	No	filter by execution status
--difficulty	No	filter by benchmark difficulty
--category	No	filter by benchmark category
--workflow-name	No	filter by workflow name
--prompt-bundle-name	No	filter by prompt bundle name

Subcommand-specific arguments:

Subcommand	Arguments	Description
summary	--output	optional JSON output path
export-csv	--output	required CSV output path
matrix	--rows / --cols / --metric / --output	matrix grouping, me