Celero

C++ Benchmarking Library

Copyright 2017-2026 John Farrier

Apache 2.0 License

Community Support

A Special Thanks to the following corporations for their support:

• Polyrhythm Software
• GitKraken
• Microsoft

Builds and Testing

As of v2.7, Celero requires the developer to provide GoogleTest in order to build unit tests. We suggest using a package manager such as VCPKG or Conan to provide the latest version of the library.

Overview

Developing consistent and meaningful benchmark results for code is a complicated task. Measurement tools exist (Intel® VTune™ Amplifier, SmartBear AQTime, Valgrind, etc.) external to applications, but they are sometimes expensive for small teams or cumbersome to utilize. This project, Celero, aims to be a small library that can be added to a C++ project and perform benchmarks on code in a way that is easy to reproduce, share, and compare among individual runs, developers, or projects. Celero uses a framework similar to that of GoogleTest to make its API more natural to use and integrate into a project.

Make automated benchmarking as much a part of your development process as automated testing.

Celero uses CMake to provide cross-platform builds. It does require a modern C++ compiler supporting C++14.

Once Celero is added to your project, you can create dedicated benchmark projects and source files. For convenience, there is a single header file and a CELERO_MAIN macro that can be used to provide a main() for your benchmark project that will automatically execute all of your benchmark tests.

Key Features

• Supports Windows, Linux, and OSX using C++14.
• The timing utilities can be used directly in production code (independent of benchmarks).
• Automatically tracks RAM usage during the experiments
• Console table output is formatted as Markdown to easily copy/paste into documents.
• Archive results to track performance over time.
• Integrates into CI/CT/CD environments with JUnit-formatted output.
• User-defined Experiment Values can scale test results, sample sizes, and user-defined properties for each run.
• User-defined Measurements allow for measuring anything in addition to timing.
• Supports Test Fixtures.
• Supports fixed-time benchmark baselines.
• Capture a rich set of timing statistics to a file.
• Easily installed using CMake, Conan, or VCPkg.

Command Line

<celeroOutputExecutable> [-g groupNameToRun] [-t resultsTable.csv] [-j junitOutputFile.xml] [-a resultArchive.csv] [-d numberOfIterationsPerDistribution] [-h]

• -g Use this option to run only one benchmark group out of all benchmarks contained within a test executable.
• -t Writes all results to a CSV file. Very useful when using problem sets to graph performance.
• -j Writes JUnit formatted XML output. To utilize JUnit output, benchmarks must use the _TEST version of the macros and specify an expected baseline multiple. When the test exceeds this multiple, the JUnit output will indicate a failure.
• -a Builds or updates an archive of historical results, tracking current, best, and worst results for each benchmark.
• -d (Experimental) builds a plot of four different sample sizes to investigate the distribution of sample results.

Celero Basics

Background

The goal, generally, of writing benchmarks is to measure the performance of a piece of code. Benchmarks are useful for comparing multiple solutions to the same problem to select the most appropriate one. Other times, benchmarks can highlight the performance impact of design or algorithm changes and quantify them in a meaningful way.

By measuring code performance, you eliminate errors in your assumptions about what the "right" solution is for performance. Only through measurement can you confirm that using a lookup table, for example, is faster than computing a value. Such lore (which is often repeated) can lead to bad design decisions and, ultimately, slower code.

The goal of writing correct benchmarking code is to eliminate all of the noise and overhead and measure only the code under test. Sources of noise in the measurements include clock resolution noise, operating system background operations, test setup/teardown, framework overhead, and other unrelated system activity.

At a theoretical level, we want to measure t, the time to execute the code under test. In reality, we measure t plus all of this measurement noise.

These extraneous contributors to our measurement of t fluctuate over time. Therefore, we want to try to isolate t. This is accomplished by making many measurements but only keeping the smallest total. The smallest total is necessarily the one with the smallest noise contribution and closest to the actual time t.

Once this measurement is obtained, it has little meaning in isolation. It is essential to create a baseline test by which to compare. A baseline should generally be a "classic" or "pure" solution to the problem on which you measure a solution. Once you have a baseline, you have a meaningful time to compare your algorithm against. Merely saying that your fancy sorting algorithm (fSort) sorted a million elements in 10 milliseconds is not sufficient by itself. However, compared to a classic sorting algorithm baseline such as quicksort (qSort), you can say that fSort is 50% faster than qSort on a million elements. That is a meaningful and powerful measurement.

Implementation

Celero heavily utilizes C++14 features that are available in modern compilers. C++14 greatly aided in making the code clean and portable. To make adopting the code more manageable, all definitions needed by a user are defined in a celero namespace within a single include file: Celero.h.

Celero.h has within it the macro definitions that turn each of the user benchmark cases into its own unique class with the associated test fixture (if any) and then registers the test case within a Factory. The macros automatically associate baseline test cases with their associated test benchmarks so that, at run time, benchmark-relative numbers can be computed. This association is maintained by TestVector.

The TestVector utilizes the PImpl idiom to help hide implementation and keep the #include overhead of Celero.h to a minimum.

Celero reports its outputs to the command line. Since colors are nice (and perhaps contribute to the human factors/readability of the results), something beyond std::cout was called for. Console.h defines a simple color function, SetConsoleColor, which is utilized by the functions in the celero::print namespace to format the program's output nicely.

Measuring benchmark execution time takes place in the TestFixture base class, from which all benchmarks are written are ultimately derived. First, the test fixture setup code is executed. Then, the start time for the test is retrieved and stored in microseconds using an unsigned long. This is done to reduce floating point error. Next, the specified number of operations (iterations) is executed. When complete, the end time is retrieved, the test fixture is torn down, and the measured time for the execution is returned, and the results are saved.

This cycle is repeated for however-many samples were specified. If no samples were specified (zero), then the test is repeated until it is run for at least one second or at least 30 samples have been taken. While writing this specific part of the code, there was a definite "if-else" relationship. However, the bulk of the code was repeated within the if and else sections. An old-fashioned function could have been used here, but utilizing std::function to define a lambda that could be called and keep all of the code clean was very natural. (C++11 is a fantastic thing.) Finally, the results are printed on the screen.

General Program Flow

To summarize, this pseudo-code illustrates how the tests are executed internally:

for(Each Experiment)
{
    for(Each Sample)
    {
        // Call the virtual function
        // and DO NOT include its time in the measurement.
        experiment->setUp();

        // Start the Timer
        timer->start();

        // Run all iterations
        for(Each Iteration)
        {
            // Call the virtual function
            // and include its time in the measurement.
            experiment->onExperimentStart(x);

            // Run the code under test
            experiment->run(threads, iterations, experimentValue);

            // Call the virtual function
            // and include its time in the measurement.
            experiment->onExperimentEnd();
        }

        // Stop the Timer
        timer->stop();

        // Record data...

        // Call the virtual teardown function
        // and DO NOT include its time in the measurement.
        experiment->tearDown();
    }
}

Using the Code

Celero uses CMake to provide cross-platform builds. It does require a modern compiler supporting C++14.

Once Celero is added to your project, you can create dedicated benchmark projects and source files. For convenience, there is a single header file and a CELERO_MAIN macro that can be used to provide a main() for your benchmark project that will automatically execute all of your benchmark tests.

Here is an example of a simple Celero Benchmark. (Note: This is a complete, runnable example.)

#include <celero/Celero.h>

#include <random>

#ifndef _WIN32
#include <cmath>
#include <cstdlib>
#endif

///
/// This is the main(int argc, char** argv) for the entire celero program.
/// You can write your own, or use this macro to insert the standard one into the project.
///
CELERO_MAIN

std::random_device RandomDevice;
std::uniform_int_distribution<int> UniformDistribution(0, 1024);

///
/// In reality, all of the "Complex" cases take the same amount of time to run.
/// The difference in the results is a product of measurement error.
///
/// Interestingly, taking the sin of a constant number here resulted in a
/// great deal of optimization in clang and gcc.
///
BASELINE(DemoSimple, Baseline, 10, 1000000)
{
    celero::DoNotOptimizeAway(static_cast<float>(sin(UniformDistribution(RandomDevice))));
}

///
/// Run a test consisting of 1 sample of 710000 operations per measurement.
/// There are not enough samples here to likely get a meaningful result.
///
BENCHMARK(DemoSimple, Complex1, 1, 710000)
{
    celero::DoNotOptimizeAway(static_cast<float>(sin(fmod(UniformDistribution(RandomDevice), 3.14159265))));
}

///
/// Run a test consisting of 30 samples of 710000 operations per measurement.
/// There are not enough samples here to get a reasonable measurement
/// It should get a Baseline number lower than the previous test.
///
BENCHMARK(DemoSimple, Complex2, 30, 710000)
{
    celero::DoNotOptimizeAway(static_cast<float>(sin(fmod(UniformDistribution(RandomDevice), 3.14159265))));
}

///
/// Run a test consisting of 60 samples of 710000 operations per measurement.
/// There are not enough samples here to get a reasonable measurement
/// It should get a Baseline number lower than the previous test.
///
BENCHMARK(DemoSimple, Complex3, 60, 710000)
{
    celero::DoNotOptimizeAway(static_cast<float>(sin(fmod(UniformDistribution(RandomDevice), 3.14159265))));
}

The first thing we do in this code is to define a BASELINE test case. This template takes four arguments:

BASELINE(GroupName, BaselineName, Samples, Operations)

• GroupName - The name of the benchmark group. This is used to batch together runs and results with their corresponding baseline measurement.
• BaselineName - The name of this baseline for reporting purposes.
• Samples - The total number of times you want to execute the given number of operations on the test code.
• Operations - The total number of times you want to run the test code per sample.

Samples and operations here are used to measure very fast code. If you know the code in your benchmark will take some time less than 100 milliseconds, for example, your operations number would say to execute the code "operations" number of times before taking a measurement. Samples define how many measurements to make.

Celero helps with this by allowing you to specify zero samples. Zero samples will tell Celero to make some statistically significant number of samples based on how long it takes to complete your specified number of operations. These numbers will be reported at run time.

The celero::DoNotOptimizeAway template is provided to ensure that the optimizing compiler does not eliminate your function or code. Since this feature is used in all of the sample benchmarks and their baseline, its time overhead is canceled out in the comparisons.

After the baseline is defined, various benchmarks are then defined. The syntax for the BENCHMARK macro is identical to that of the macro.

Results

Running Celero's simple example experiment (celeroDemoSimple.exe) benchmark gave the following output on a PC:

Celero
Timer resolution: 0.277056 us
|   Group    | Experiment | Prob. Space | Samples | Iterations | Baseline | us/Iteration | Iterations/sec | RAM (bytes) |
| :--------: | :--------: | :---------: | :-----: | :--------: | :------: | :----------: | :------------: | :---------: |
| DemoSimple |  Baseline  |    Null     |   30    |  1000000   | 1.00000  |   0.09320    |  10729498.61   |   892928    |
| DemoSimple |  Complex1  |    Null     |    1    |   710000   | 0.99833  |   0.09305    |  10747479.64   |   897024    |
| DemoSimple |  Complex2  |    Null     |   30    |   710000   | 0.97898  |   0.09124    |  10959834.52   |   897024    |
| DemoSimple |  Complex3  |    Null     |   60    |   710000   | 0.98547  |   0.09185    |  10887733.66   |   897024    |
Completed in 00:00:10.315012

The first test t