Phantora

Estimate the throughput, MFU, and iteration time of distributed LLM training on clusters you don't own — by running a Megatron, DeepSpeed, or TorchTitan script on a single GPU.

Phantora is a hybrid GPU cluster simulator for ML system performance estimation. Instead of asking you to reimplement your workload in a simulator's DSL, Phantora intercepts the GPU and NCCL calls of an ML framework, simulates them, and lets the framework's own performance-logging code (e.g., mfu, iteration time) print as if you ran on a real cluster.

Phantora was accepted at NSDI 2026 🎉 — see the paper for design details.

How it works

You add a few lines to your training script to enable the Phantora tracer, then run it on a single GPU. A stub libcuda.so and libnccl.so intercept every GPU and collective call and forward them over a Unix socket to a Rust simulator (phantora_server), which:

• estimates each CUDA kernel's execution time from a one-time profile on the local GPU,
• simulates NCCL collectives over a flow-level network simulator with a configurable cluster topology,
• maintains a per-rank virtual clock and event queue, and
• patches time.perf_counter (and a few framework hooks) so the framework sees simulated time.

The framework's own logging then emits the metrics it would on a real cluster — produced from a single GPU and a virtual cluster configuration of your choosing.

What Phantora can simulate

Phantora runs your unmodified Megatron-LM, DeepSpeed, or TorchTitan training script and estimates how it would perform on a virtual cluster you describe — any GPU count, per-GPU VRAM, and network topology — all from a single GPU.

Available model presets

Ready-to-run presets ship for each framework. ✅ links to the preset — a launcher script for Megatron/DeepSpeed (under tests/docker/<framework>/), or a .toml config for TorchTitan; — means there is no preset for that pair (the model may still be expressible by hand). All MoE presets assume load-balanced experts (why).

1. TorchTitan flavors are fixed-size; run with --training.debug_moe_force_load_balance. The full 30B-A3B flavor targets a real multi-GPU cluster; for a quick check on a modest box, add --model.flavor=debugmodel_moe.

See Try our examples for how to launch one.

Parallelism support

The strategies Phantora models today, which compose (e.g. TP+EP with sequence parallelism, or DP+PP):

✅ = simulated end-to-end; — = the strategy does not exist in that framework. ¹ MoE is supported under a load-balanced-experts assumption (see Limitations).

Phantora's estimates have been validated against real-hardware ground truth to within a few percent — see Accuracy: Validated Configurations.

Requirements

• Linux x86_64
• An NVIDIA GPU for kernel profiling. The build targets compute capability 8.0 (e.g., A100, H200) and 9.0 (e.g., H100); other GPUs may work but are untested.
• CUDA 12.8 (the Docker image is based on nvidia/cuda:12.8.0-devel-ubuntu22.04)
• Docker with Docker Compose (recommended)
• Python 3.11.9 if building outside Docker
• Tens of GB of free disk for the image and downloaded model assets

Build Instructions

Clone the repository via git.

git clone https://github.com/QDelta/Phantora
cd Phantora
git submodule update --init --recursive

Note: pytorch/ is a git submodule pointing at a custom PyTorch branch (2.9.1-phantora) with the function tracer patched.

Docker (with Docker Compose) is recommended for building and using Phantora. In the repository root, run:

docker build -t phantora .

It might take a while.

If you want to build it locally without Docker, also refer to Dockerfile for the detailed commands.

Try our examples

Once you built the phantora docker image, you can try our examples of distributed training using Megatron, DeepSpeed and TorchTitan. The examples will launch multiple containers (using Docker Compose) to simulate a GPU cluster.

For example, to simulate a distributed Llama2 7B training using Megatron:

cd tests/docker/megatron

# Generate configurations for a 16-GPU cluster with 140GB VRAM per GPU
python3 config_gen.py --nhost 4 --ngpu 4 --vram_mib 143771

# Start training
./run.sh

# ... look at the terminal output

# Cleanup containers and other temporary files
./stop.sh

Similar for DeepSpeed and TorchTitan.

TorchTitan (≥ 0.2.0) loads a Hugging Face tokenizer directory (tokenizer.json + config) via hf_assets_path, not a tiktoken .model file. Place any HF tokenizer in tests/assets/hf_tokenizer/ before starting — under payload-free simulation only its vocab size matters (it sets the embedding dimensions). The Llama3 tokenizer works, or any ungated one (e.g. openai-community/gpt2).

run.sh will pass its arguments to the corresponding scripts (tests/test_{megatron,deepspeed,torchtitan}.py). Each entry in the model-preset table links to a ready-made launcher you can run this way.

Run without a GPU (performance database)

Phantora normally needs one GPU to profile each kernel's time. For the model presets, those timings can be recorded once and replayed, so the presets can be simulated on a machine with no GPU and no NVIDIA driver — only the CUDA libraries bundled in the Phantora image are present (and never called in replay; verified by replaying every preset in containers with no driver injected).

A preset produces a fixed, enumerable set of kernel shapes, so a recorded performance database (tests/perfdb/<gpu>/, plain CSV) is complete for it. In replay the simulator answers every timing query from the database instead of touching the GPU.

Replay a preset (no GPU)

--perf-db <name> makes config_gen.py point the simulator at tests/perfdb/<name>/ and drop the simulator's GPU reservation, so the whole stack runs GPU-free:

cd tests/docker/megatron
python3 config_gen.py --nhost 1 --ngpu 2 --vram_mib 81920 --perf-db l40s
./mixtral/run_mixtral_8x7b.sh --expert_model_parallel_size 2 --sequence_length 1024 --num_layers 4

The committed tests/perfdb/l40s/ (recorded on an NVIDIA L40S) covers each preset at the exact config tests/perfdb/record_all.sh recorded it at — for Mixtral that is the 2-GPU, EP=2, sequence-length-1024 config above, which is why the command differs from the 8-GPU one in the preset's own header. Replaying a different config (more GPUs, a longer sequence, a different micro-batch) introduces kernel shapes the database does not have; that is not an error, but the run's numbers are invalid until you complete the database — see the next section. The CSV is human-readable — each row is one (op, shape) → nanoseconds entry.

A database is specific to the GPU and to the run config (parallelism, sequence length, micro-batch), but not to num_layers for key coverage: every layer repeats the same kernel shapes, so a 4-layer recording's keys also cover the full-depth model. Memory still scales with depth, though — replaying at full depth needs a --vram_mib large enough to hold it, and full-depth Mixtral in particular only fits with the expert parallelism its preset header assumes (EP=8), a config the committed database does not cover. The --num_layers 4 replay above sidesteps that while still exercising the complete kernel set GPU-free.

Changing the config (e.g. a larger context window)

If you change a parameter that introduces new kernel shapes — say a bigger --sequence_length — the database won't have them. Phantora doesn't crash: it finishes the run (charging the unknown kernels zero time, so those numbers are invalid) and writes the exact list of missing shapes to tests/perfdb/<name>.missing/, with a message telling you what to do. To complete the database you profile just those shapes on a GPU and merge them back:

# On any GPU machine (no Phantora build needed — just PyTorch):
python3 tests/perfdb/bench.py --ref tests/perfdb/l40s.missing --out tests/perfdb/l40s --merge
# then re-run the replay above — now complete, still no GPU on your machine.

This is the contribution loop: GPU-less users discover the shapes they need (no GPU), the few new timings get profiled once on any GPU, and because the database is plain CSV it's a clean pull request — so over time the common configs are already covered and nobody needs a GPU.

Recording / regenerating a database (on a GPU)

# Record a config while running it on a GPU (profiles, then writes/merges tests/perfdb/<NAME>/):
python3 config_gen.py --nhost 1 --ngpu 8 --vram_mib 81920 --record-perf-db l40s
./run.sh ...
# or record every preset at once:
tests/perfdb/record_all.sh <NAME>

Recording merges into an existing database: shapes it already contains are not re-profiled, so a re-record only adds the new ones. To re-time entries that are already there (e.g. after a change to how a kernel is captured), delete the directory first and record it from scratch. Recording into a database captured on a different GPU is refused, since it would leave the old GPU's timings in place while relabelling the database with the new GPU's name.

To build a database for a different GPU without building Phantora at all, tests/perfdb/bench.py re-profiles an existing database's shapes using only stock PyTorch (it reads the (op, shape) keys and re-times each kernel locally):

python3 tests/perfdb/bench.py --ref tests/perfdb/l40s   # writes tests/perfdb/<your-gpu>/

It mirrors Phantora's profiler (kernel-only timing, operand aliasing) and reproduces a Phantora-recorded database to ~1% per-op, giving an identical simulated iteration time.

Adapt your training scripts

Scripts and configurations in tests/ will be good examples.

Generally, edit your script like this:

from phantora_utils import (
    enable_function_tracer,
    disable_function_tracer,
)

# ... Your original script
# Use time.perf_counter or phantora_utils.time for timers

if __name__ == "__main__":
    enable_function_tracer()
    # ... Your original main
    disable_function_tracer()

For other configurations, you can refer to generated configurations in tests/docker/{megatron,deepspeed,torchtitan}.

Accuracy: Validated Configurations

The tables below list the configurations we have validated against real-hardware ground truth. We would love community contributions of additional ground-truth measurements so that we can better understand Phantora's accuracy across hardware, frameworks, and workloads. See Contributing ground truth below.

Hardware

Phantora itself always runs on a single GPU. The three host configurations we used are:

Ground truth comes from a mix of in-house testbeds and published reports:

• Megatron Llama2 7B — in-house ground truth on the same physical box as the H200 host, using all 4× H200 NVL GPUs over NVLink.
• TorchTitan (FSDP2) — TorchTitan's published H100 and A100-80G performance reports. H100 targets are simulated on the H200 host; A100-80G targets are simulated on the A100 host. In both cases Phantora's VRAM is configured to 80 GB to match the target.
• DeepSpeed non-LLM workloads — in-house 4-server, 8-GPU RTX 3090 cluster (2 GPUs per server over Ethernet).

Megatron — Llama2 7B

Comparison against on-testbed measurements, with and without optimizer.

Reported accuracy: average error 3.7%, maximum **