Pre-allocated memory pool that provides unique ``data_ptr`` per iteration. Allocates a buffer only slightly larger than the input tensors (ov

Concrete workload configuration for benchmarking. Defines a specific instance of a computational workload with concrete values for all variab

Configuration for benchmark runs. All fields have default values to make configuration optional.

Complete trace linking a solution to a definition with evaluation results. A Trace represents the complete record of benchmarking a specific solu

Parameters for one example e2e test case.

A concrete implementation for a given Definition. Represents a complete solution that provides a high-performance implementation for a comput

Complete definition of a computational workload. A Definition provides a formal, machine-readable specification for a computational workload.

GPU and DRAM clock frequencies for stable benchmarking.

Correctness metrics from numerical evaluation. Contains error measurements comparing the solution output against a reference implementation t

Stage files for compilation and execution, returning commands for the CLI.

Raised when a reward-hacking pattern is detected in a submission.

Supported programming languages for solution implementations. Enumeration of programming languages that can be used to implement solutions fo

Parameters for one reward-hack e2e test case.

Parameters for one SOL ExecBench e2e test case.

Tolerance specification for a workload. Specifies the conditions for a particular problem to be deemed numerically correct against the reference

Build specification for a solution implementation. Contains all technical specifications required to build and execute a solution, including

Environment information from evaluation execution. Records the hardware and software environment details from when the evaluation was perform

Complete evaluation result for a solution on a workload. Records the full outcome of benchmarking a solution implementation against a specifi

Performance metrics from timing evaluation. Contains timing measurements and performance comparisons from benchmarking the solution against r

Constant axis with a fixed value. A constant axis represents a dimension that has a fixed, compile-time known value. This is useful for dimen

Expression axis that can be specified at runtime. An expression axis represents a dimension whose value is determined at runtime based on a m

Variable axis that can be specified at runtime. A variable axis represents a dimension whose value is determined at runtime based on the actu

Base model with the attribute docstrings being extracted to the model JSON schema.

Compiler and linker flags for C++/CUDA solutions. Passed directly to the underlying build system (e.g. torch.utils.cpp_extension.load).

Custom input specification. Represents an that is generated from custom reference code. If a single input is this type, all inputs should be this

Supported data types for tensors. Enumeration of all data types that can be used in tensor specifications. Includes both floating-point and i

Status codes for evaluation results. Enumeration of all possible outcomes when evaluating a solution against a workload, covering success and

Random input generation descriptor. Represents a specification for generating random tensor input data during workload execution and benchmar

Input specification for data loaded from safetensors files. Represents tensor data that will be loaded from a safetensors file using a specif

Scalar literal input specification. Represents a scalar value (integer, float, or boolean) that will be used as a direct input parameter to t

A single source code file in a solution implementation. Represents a source code file with its relative path and complete content. The file c

Supported bindings for C++/CUDA solution implementations. Enumeration of binding types that can be used to interface compiled C++/CUDA code w

Supported hardware targets for solution implementations. Enumeration of hardware platforms that solutions can target.

Specification for a tensor including shape and data type, to use as input or output of a kernel. This includes the symbolic shape (referencin

GPU integration tests for CUPTI activity tracing timing accuracy. These tests mirror TestBenchTimeWithCUDAEventsGPU but use bench_gpu_time_wi

GPU integration tests for do_bench timing accuracy. These tests verify that CUDA event timing correctly captures kernel execution time while

Tests for compile option extraction.

Tests for compute_error_stats, focusing on near-zero edge cases.

CUPTI should report lower or equal median times vs CUDA events. CUDA events include a fixed ~4-6us overhead from event recording. For short k

Tests for dtype_str_to_python_dtype.

Tests for dtype_str_to_torch_dtype (canonical source in data/dtypes.py).

BuildSpec must reject entry points whose suffix doesn't match the language category.

Tests for ext.load call arguments.

Tests for extra_include_paths construction.

Tests for is_dtype_integer.

BuildSpec must reject specs that mix C++ and Python languages.

Tests for lock_clocks(). verify_clocks() and time.sleep() are mocked so we only test the locking logic. Verification is tested separately in

Tests that set_seed produces reproducible GPU tensor generation.

Tests for the .so rename logic after compilation.

Tests for source file collection logic.

CUPTI-based timing must capture work hidden on non-default streams. A submission could launch kernels on a non-default stream to evade CUDA e

Verify the raw template is valid Python.