mathbench

mathbench is a suite of unit tests and benchmarks comparing the output and performance of a number of different Rust linear algebra libraries for common game and graphics development tasks.

mathbench is written by the author of glam and has been used to compare the performance of glam with other similar 3D math libraries targeting games and graphics development, including:

• cgmath
• euclid
• nalgebra
• pathfinder_geometry
• static-math
• ultraviolet
• vek

The benchmarks

All benchmarks are performed using Criterion.rs. Benchmarks are logically into the following categories:

• return self - attempts to measure overhead of benchmarking each type.
• single operations - measure the performance of single common operations on types, e.g. a matrix inverse, vector normalization or multiplying two matrices.
• throughput operations - measure the performance of common operations on batches of data. These measure operations that would commonly be processing batches of input, for example transforming a number of vectors with the same matrix.
• workload operations - these attempt to recreate common workloads found in game development to try and demonstrate performance on real world tasks.

Despite best attempts, take the results of micro benchmarks with a pinch of salt.

Operation benchmarks

• matrix benches - performs common matrix operations such as transpose, inverse, determinant and multiply.
• rotation 3d benches - perform common 3D rotation operations.
• transform 2d & 3d benches - bench special purpose 2D and 3D transform types. These can be compared to 3x3 and 4x4 matrix benches to some extent.
• transformations benches - performs affine transformations on vectors - uses the best available type for the job, either matrix or transform types depending on the library.
• vector benches - perform common vector operations.

Workload benchmarks

• euler bench - performs an Euler integration on arrays of 2D and 3D vectors

The benchmarks are currently focused on f32 types as that is all glam currently supports.

Crate differences

Different libraries have different features and different ways of achieving the same goal. For the purpose of trying to get a performance comparison sometimes mathbench compares similar functionality, but sometimes it's not exactly the same. Below is a list of differences between libraries that are notable for performance comparisons.

Matrices versus transforms

The euclid library does not support generic square matrix types like the other libraries tested. Rather it has 2D and 3D transform types which can transform 2D and 3D vector and point types. Each library has different types for supporting transforms but euclid is unique amongst the libraries tested in that is doesn't have generic square matrix types.

The Transform2D is stored as a 3x2 row major matrix that can be used to transform 2D vectors and points.

Similarly Transform3D is used for transforming 3D vectors and points. This is represented as a 4x4 matrix so it is more directly comparable to the other libraries however it doesn't support some operations like transpose.

There is no equivalent to a 2x2 matrix type in euclid.

Matrix inverse

Note that cgmath and nalgebra matrix inverse methods return an Option whereas glam and euclid do not. If a non-invertible matrix is inverted by glam or euclid the result will be invalid (it will contain NaNs).

Quaternions versus rotors

Most libraries provide quaternions for performing rotations except for ultraviolet which provides rotors.

Wide benchmarks

All benchmarks are gated as either "wide" or "scalar". This division allows us to more fairly compare these different styles of libraries.

"scalar" benchmarks operate on standard scalar f32 values, doing calculations on one piece of data at a time (or in the case of a "horizontal" SIMD library like glam, one Vec3/Vec4 at a time).

"wide" benchmarks operate in a "vertical" AoSoA (Array-of-Struct-of-Arrays) fashion, which is a programming model that allows the potential to more fully use the advantages of SIMD operations. However, it has the cost of making algorithm design harder, as scalar algorithms cannot be directly used by "wide" architectures. Because of this difference in algorithms, we also can't really directly compare the performance of "scalar" vs "wide" types because they don't quite do the same thing (wide types operate on multiple pieces of data at the same time).

The "wide" benchmarks still include glam, a scalar-only library, as a comparison. Even though the comparison is somewhat apples-to-oranges, in each of these cases, when running "wide" benchmark variants, glam is configured to do the exact same amount of final work, producing the same outputs that the "wide" versions would. The purpose is to give an idea of the possible throughput benefits of "wide" types compared to writing the same algorithms with a scalar type, at the cost of extra care being needed to write the algorithm.

To learn more about AoSoA architecture, see this blog post by the author of nalgebra which goes more in depth to how AoSoA works and its possible benefits. Also take a look at the "Examples" section of ultraviolet's README, which contains a discussion of how to port scalar algorithms to wide ones, with the examples of the Euler integration and ray-sphere intersection benchmarks from mathbench.

Note that the nalgebra_f32x4 and nalgebra_f32x8 benchmarks require a Rust

Additionally the f32x8 benchmarks will require the AVX2 instruction set, to enable that you will need to build with RUSTFLAGS='-C target-feature=+avx2.

Build settings

The default profile.bench settings are used, these are documented in the cargo reference.

Some math libraries are optimized to use specific instruction sets and may benefit building with settings different to the defaults. Typically a game team will need to decided on a minimum specification that they will target. Deciding on a minimum specifiction dictates the potential audience size for a project. This is an important decision for any game and it will be different for every project. mathbench doesn't want to make assumptions about what build settings any particular project may want to use which is why default settings are used.

I would encourage users who to use build settigs different to the defaults to run the benchmarks themselves and consider publishing their results.

Benchmark results

The following is a table of benchmarks produced by mathbench comparing glam performance to cgmath, nalgebra, euclid, vek, pathfinder_geometry, static-math and ultraviolet on f32 data.

These benchmarks were performed on an [Intel i7-4710HQ] CPU on Linux. They were compiled with the 1.49.0-nightly (ffa2e7ae8 2020-10-24) Rust compiler. Lower (better) numbers are highlighted within a 2.5% range of the minimum for each row.

The versions of the libraries tested were:

• cgmath - 0.17.0
• euclid - 0.22.1
• glam - 0.10.0
• nalgebra - 0.23.0
• pathfinder_geometry - 0.5.1
• static-math - 0.1.7
• ultraviolet - 0.5.1
• vek - 0.12.0 (repr_c types)

See the full [mathbench report] for more detailed results.

Scalar benchmarks

Run with the command:

cargo +nightly bench --features scalar scalar