Benchmarking nearest neighbors

Doing fast searching of nearest neighbors in high dimensional spaces is an increasingly important problem with notably few empirical attempts at comparing approaches in an objective way, despite a clear need for such to drive optimization forward.

This project contains tools to benchmark various implementations of approximate nearest neighbor (ANN) search for selected metrics. We have pre-generated datasets (in HDF5 format) and prepared Docker containers for each algorithm, as well as a test suite to verify function integrity.

Evaluated

• Annoy
• FLANN
• scikit-learn: LSHForest, KDTree, BallTree
• Weaviate
• PANNS
• NearPy
• KGraph
• NMSLIB (Non-Metric Space Library) : SWGraph, HNSW, BallTree, MPLSH
• hnswlib (a part of nmslib project)
• RPForest
• FAISS
• DolphinnPy
• Datasketch
• nndescent
• PyNNDescent
• MRPT
• NGT : ONNG, PANNG, QG
• SPTAG
• PUFFINN
• N2
• ScaNN
• Vearch
• Elasticsearch : HNSW
• Elastiknn
• ExpANN
• OpenSearch KNN
• DiskANN : Vamana, Vamana-PQ
• Vespa
• scipy: cKDTree
• vald
• Qdrant
• HUAWEI(qsgngt)
• Milvus : Knowhere
• Zilliz(Glass)
• pgvector
• pgvecto.rs
• RediSearch
• pg_embedding
• Descartes(01AI)
• kgn
• vsag
• PGVectorScale

Data sets

We have a number of precomputed data sets in HDF5 format. All data sets have been pre-split into train/test and include ground truth data for the top-100 nearest neighbors.

Results

These are all as of April 2025, running all benchmarks on a r6i.16xlarge machine on AWS with --parallelism 31 and hyperthreading disabled. All benchmarks are single-CPU.

glove-100-angular

sift-128-euclidean

fashion-mnist-784-euclidean

nytimes-256-angular

gist-960-euclidean

glove-25-angular

TODO: update plots on http://ann-benchmarks.com.

Install

The only prerequisite is Python (tested with 3.10.6) and Docker.

1. Clone the repo.
2. Run pip install -r requirements.txt.
3. Run python install.py to build all the libraries inside Docker containers (this can take a while, like 10-30 minutes).

Running

1. Run python run.py (this can take an extremely long time, potentially days).
2. Run python plot.py --x-scale logit --y-scale log to plot results.
3. Run python create_website.py to create a website with lots of plots.

You can customize the algorithms and datasets as follows:

• Check that ann_benchmarks/algorithms/{YOUR_IMPLEMENTATION}/config.yml contains the parameter settings that you want to test
• To run experiments on SIFT, invoke python run.py --dataset glove-100-angular. See python run.py --help for more information on possible settings. Note that experiments can take a long time.
• To process the results, either use python plot.py --dataset glove-100-angular or python create_website.py. An example call: python create_website.py --plottype recall/time --latex --scatter --outputdir website/.

Including your algorithm

Add your algorithm in the folder `ann