RoaringBitmap

[![][license img]][license] [![docs-badge][]][docs]

• Introduction
• When should you use a bitmap?
• When should you use compressed bitmaps?
• How does Roaring compare with the alternatives?
• Code sample
• API docs
• Download
• Usage within a Maven project
• Usage within a gradle project
• Scientific Documentation
• Unsigned integers
• Working with memory-mapped bitmaps
• Thread safety
• Kryo
• 64-bit integers (long)
• Range Bitmaps
• COW Roaring Bitmaps
• Prerequisites
• Usage for RoaringBitmap Developers
• IntelliJ and Eclipse
• Contributing
• FAQ
• Benchmark
• Mailing list/discussion group
• Funding

Introduction

Bitsets, also called bitmaps, are commonly used as fast data structures. Unfortunately, they can use too much memory. To compensate, we often use compressed bitmaps.

Roaring bitmaps are compressed bitmaps which tend to outperform conventional compressed bitmaps such as WAH, EWAH or Concise. In some instances, roaring bitmaps can be hundreds of times faster and they often offer significantly better compression. They can even be faster than uncompressed bitmaps.

Roaring bitmaps are found to work well in many important applications:

Use Roaring for bitmap compression whenever possible. Do not use other bitmap compression methods (Wang et al., SIGMOD 2017)

kudos for making something that makes my software run 5x faster (Charles Parker from BigML)

This library is used by * Apache Spark, * Apache Hive, * Apache Tez, * Apache Kylin, * Apache CarbonData, * Netflix Atlas, * OpenSearchServer, * zenvisage, * Jive Miru, * Tablesaw, * Apache Hivemall, * Gaffer, * Apache Pinot, * Apache Druid, * SirixDB * EvitaDB * Apache Iceberg * Apache Pulsar

The library is mature and has been used in production for many years.

The YouTube SQL Engine, Google Procella, uses Roaring bitmaps for indexing. Apache Lucene uses Roaring bitmaps, though they have their own independent implementation. Derivatives of Lucene such as Solr and Elastic also use Roaring bitmaps. Other platforms such as Whoosh, Microsoft Visual Studio Team Services (VSTS) and Pilosa also use Roaring bitmaps with their own implementations. You find Roaring bitmaps in InfluxDB, Bleve, Cloud Torrent, Redpanda, and so forth.

There is a serialized format specification for interoperability between implementations. We have interoperable C/C++, Java and Go implementations.

(c) 2013-... the RoaringBitmap authors

This code is licensed under Apache License, Version 2.0 (AL2.0).

When should you use a bitmap?

Sets are a fundamental abstraction in software. They can be implemented in various ways, as hash sets, as trees, and so forth. In databases and search engines, sets are often an integral part of indexes. For example, we may need to maintain a set of all documents or rows (represented by numerical identifier) that satisfy some property. Besides adding or removing elements from the set, we need fast functions to compute the intersection, the union, the difference between sets, and so on.

To implement a set of integers, a particularly appealing strategy is the bitmap (also called bitset or bit vector). Using n bits, we can represent any set made of the integers from the range [0,n): the ith bit is set to one if integer i is present in the set. Commodity processors use words of W=32 or W=64 bits. By combining many such words, we can support large values of n. Intersections, unions and differences can then be implemented as bitwise AND, OR and ANDNOT operations. More complicated set functions can also be implemented as bitwise operations.

When the bitset approach is applicable, it can be orders of magnitude faster than other possible implementation of a set (e.g., as a hash set) while using several times less memory.

However, a bitset, even a compressed one is not always applicable. For example, if you have 1000 random-looking integers, then a simple array might be the best representation. We refer to this case as the "sparse" scenario.

When should you use compressed bitmaps?

An uncompressed BitSet can use a lot of memory. For example, if you take a BitSet and set the bit at position 1,000,000 to true and you have just over 100kB. That is over 100kB to store the position of one bit. This is wasteful even if you do not care about memory: suppose that you need to compute the intersection between this BitSet and another one that has a bit at position 1,000,001 to true, then you need to go through all these zeroes, whether you like it or not. That can become very wasteful.

This being said, there are definitively cases where attempting to use compressed bitmaps is wasteful. For example, if you have a small universe size. E.g., your bitmaps represent sets of integers from [0,n) where n is small (e.g., n=64 or n=128). If you can use an uncompressed BitSet and it does not blow up your memory usage, then compressed bitmaps are probably not useful to you. In fact, if you do not need compression, then a BitSet offers remarkable speed.

The sparse scenario is another use case where compressed bitmaps should not be used. Keep in mind that random-looking data is usually not compressible. E.g., if you have a small set of 32-bit random integers, it is not mathematically possible to use far less than 32 bits per integer, and attempts at compression can be counterproductive.

How does Roaring compare with the alternatives?

Most alternatives to Roaring are part of a larger family of compressed bitmaps that are run-length-encoded bitmaps. They identify long runs of 1s or 0s and they represent them with a marker word. If you have a local mix of 1s and 0, you use an uncompressed word.

There are many formats in this family:

• Oracle's BBC (Byte-aligned Bitmap Code) is an obsolete format at this point: though it may provide good compression, it is likely much slower than more recent alternatives due to excessive branching.
• WAH (Word Aligned Hybrid) is a patented variation on BBC that provides better performance.
• Concise is a variation on the patented WAH. In some specific instances, it can compress much better than WAH (up to 2x better), but it is generally slower.
• EWAH (Enhanced Word Aligned Hybrid) is both free of patent, and it is faster than all the above. On the downside, it does not compress quite as well. It is faster because it allows some form of "skipping" over uncompressed words. So though none of these formats are great at random access, EWAH is better than the alternatives.

There is a big problem with these formats however that can hurt you badly in some cases: there is no random access. If you want to check whether a given value is present in the set, you have to start from the beginning and "uncompress" the whole thing. This means that if you want to intersect a big set with a large set, you still have to uncompress the whole big set in the worst case...

Roaring solves this problem. It works in the following manner. It divides the data into chunks of 2¹⁶ integers (e.g., [0, 2¹⁶), [2¹⁶, 2 x 2¹⁶), ...). Within a chunk, it can use an uncompressed bitmap, a simple list of integers, or a list of runs. Whatever format it uses, they all allow you to check for the presence of any one value quickly (e.g., with a binary search). The net result is that Roaring can compute many operations much faster than run-length-encoded formats like WAH, EWAH, Concise... Maybe surprisingly, Roaring also generally offers better compression ratios.

Code sample

import org.roaringbitmap.RoaringBitmap;

public class Basic {

  public static void main(String[] args) {
        RoaringBitmap rr = RoaringBitmap.bitmapOf(1,2,3,1000);
        RoaringBitmap rr2 = new RoaringBitmap();
        rr2.add(4000L,4255L);
        rr.select(3); // would return the third value or 1000
        rr.rank(2); // would return the rank of 2, which is index 1
        rr.contains(1000); // will return true
        rr.contains(7); // will return false

        RoaringBitmap rror = RoaringBitmap.or(rr, rr2);// new bitmap
        rr.or(rr2); //in-place computation
        boolean equals = rror.equals(rr);// true
        if(!equals) throw new RuntimeException("bug");
        // number of values stored?
        long cardinality = rr.getLongCardinality();
        System.out.println(cardinality);
        // a "forEach" is faster than this loop, but a loop is possible:
        for(int i : rr) {
          System.out.println(i);
        }
  }
}

You can serialize and deserialize bitmaps:

    RoaringBitmap rb = new RoaringBitmap();
    for (int k = 0; k < 100000; k += 1000) {
      rb.add(k);
    }
    String file1 = "bitmapwithoutruns.bin";
    try (DataOutputStream out = new DataOutputStream(new FileOutputStream(file1))) {
      rb.serialize(out);
    }
    rb.runOptimize();
    String file2 = "bitmapwithruns.bin";
    try (DataOutputStream out = new DataOutputStream(new FileOutputStream(file2))) {
      rb.serialize(out);
    }
    // recover
    RoaringBitmap rbtest = new RoaringBitmap();
    try (DataInputStream in = new DataInputStream(new FileInputStream(file1))) {
      rbtest.deserialize(in);
      if(!rbtest.validate()) throw new RuntimeException("bug!");
    }

Observe how, after calling deserialize, we call validate(): when deserializing content from untrusted sources, we recommand calling validate() to ensure that the content is a valid bitmap. Furthermore, we recommend using hashing to ensure that the content has not been tempered with.

This last examples also illustrates the use of runOptimize() which is sometimes helpful to reduce the size of the bitmaps.

Please see the examples folder for more examples, which you can run with ./gradlew :examples:runAll, or run a specific one with ./gradlew :examples:runExampleBitmap64, etc.

API docs

http://www.javadoc.io/doc/org.roaringbitmap/RoaringBitmap/

Download

You can download releases from github: https://github.com/RoaringBitmap/RoaringBitmap/releases

Usage within a Maven project

1. Using JitPack

Add the following dependency to your pom.xml file...

<dependency>
    <groupId>com.github.RoaringBitmap.RoaringBitmap</groupId>
    <artifactId>RoaringBitmap</artifactId>
    <version>1.6.9</version>
</dependency>

You may adjust the version number.

Then add the repository to your pom.xml file:

<repositories>
    <repository>
        <id>jitpack.io</id>
        <url>https://jitpack.io</url>
    </repository>
</repositories>

See https://github.com/RoaringBitmap/JitPackRoaringBitmapProject for a complete example.

2. Using Maven Central

Add the following dependency to your pom.xml file inside the <dependencies> element...

<dependency>
    <groupId>org.roaringbitmap</groupId>
    <artifactId>RoaringBitmap</artifactId>
    <version>1.6.9</version>
</dependency>
<dependency>
    <groupId>org.roaringbitmap</groupId>
    <artifactId>bsi</artifactId>
    <version>1.6.9</version>
</dependency>

See https://github.com/RoaringBitmap/CentralRoaringBitmapProject for a complete example.

Usage within a gradle project

1. Using JitPack

Then all you need is to edit your build.gradle file like so:

plugins {
    id 'java'
}

group 'my.awesome.project' // name of your project
version '1.0-SNAPSHOT' // version of your project

repositories {
    mavenCentral()
    maven {
        url 'https://jitpack.io'
    }
}

dependencies {
    implementation 'org.roaringbitmap:RoaringBitmap:1.6.9'
    testImplementation 'junit:junit:3.8.1'
}

See https://github.com/RoaringBitmap/JitPackRoaringBitmapProject for a complete example.

2. Using Maven Central

All you need is to edit your build.gradle file like so:

```groovy plugins { id 'java' }

group 'me.project' // name of your project version '1.0-SNAPSHOT' // version of your project

re