Reindexer

Reindexer is an embeddable, in-memory, document-oriented database with a high-level Query builder interface.

Reindexer's goal is to provide fast search with complex queries. We at Restream weren't happy with Elasticsearch and created Reindexer as a more performant alternative.

The core is written in C++ and the application level API is in Go.

This document describes the Go connector and its API. For information about the Reindexer server and HTTP API, refer to the Reindexer documentation.

Table of contents:

• Features
• Performance
• Memory Consumption
• Full text search
• Vector indexes (ANN/KNN)
• Hybrid search
• Disk Storage
• Replication
• Sharding
• Usage
• SQL compatible interface
• Installation
• Installation for server mode
  • Official docker image
• Installation for embedded mode
  • Prerequisites
  • Get Reindexer using go.mod
  • Get Reindexer using go.mod and replace
  • Get Reindexer for apps without go.mod (vendoring)
  • Get Reindexer using go.mod (vendoring)
• Advanced Usage
• Index Types and Their Capabilities
  • NULL-values filtration
• Nested Structs
• Sort
  • Forced sort
• Functions
  • flat_array_len(field_name)
  • now(unit)
• Counting
• Text pattern search with LIKE condition
• Update queries
  • Update queries with inner joins and subqueries
  • Update field with object
  • Remove field via update-query
  • Update array elements by indexes
  • Concatenate arrays
  • Remove array elements by values
• Delete queries
• Truncate queries
• Transactions and batch update
  • Synchronous mode
  • Async batch mode
  • Transactions commit strategies
  • Implementation notes
• Join
  • Anti-join
  • Joinable interface
• Subqueries (nested queries)
• Complex Primary Keys and Composite Indexes
• Aggregations
• Search in array fields
  • Search in array fields with matching indexes
  • Search in array fields with matching indexes using grouping
  • Query Execution Examples
  • Grouped values extraction examples for more complex cases
• Atomic on update functions
• Expire Data from Namespace by Setting TTL
• Direct JSON operations
  • Upsert data in JSON format
  • Get Query results in JSON format
• Using object cache
  • DeepCopy interface
  • Get shared objects from object cache (USE WITH CAUTION)
  • Limit size of object cache
  • Geometry
• Events subscription
• Logging, debug, profiling and tracing
• Turn on logger
• Slow actions logging
• Debug queries
• Custom allocators support
• Profiling
  • Heap profiling
  • CPU profiling
  • Known profiling issues
• Tracing
• Integration with other programming languages
• Reindexer for python
• Reindexer for java
  • Spring wrapper
• 3rd party open source connectors
  • PHP
  • Rust
  • .NET
• Limitations and known issues
• Getting help
• References

Features

Key features:

• Sortable indices
• Aggregation queries
• Indices on array fields
• Complex primary keys
• Composite indices
• Join operations
• Full-text search
• Up to 256 indexes (255 user's index + 1 internal index) for each namespace
• ORM-like query interface
• SQL queries

Performance

Performance has been our top priority from the start, and we think we managed to get it pretty good. Benchmarks show that Reindexer's performance is on par with a typical key-value database. On a single CPU core, we get:

• up to 500K queries/sec for queries SELECT * FROM items WHERE id='?'
• up to 50K queries/sec for queries SELECT * FROM items WHERE year > 2010 AND name = 'string' AND id IN (....)
• up to 20K queries/sec for queries SELECT * FROM items WHERE year > 2010 AND name = 'string' JOIN subitems ON ...

See benchmarking results and more details in benchmarking repo

Memory Consumption

Reindexer aims to consume as little memory as possible; most queries are processed without any memory allocation at all.

To achieve that, several optimizations are employed, both on the C++ and Go level:

• Documents and indices are stored in dense binary C++ structs, so they don't impose any load on Go's garbage collector.

• String duplicates are merged.

• Memory overhead is about 32 bytes per document + ≈4-16 bytes per each search index.

• There is an object cache on the Go level for deserialized documents produced after query execution. Future queries use pre-deserialized documents, which cuts repeated deserialization and allocation costs

• The Query interface uses sync.Pool for reusing internal structures and buffers. The combination of these technologies allows Reindexer to handle most queries without any allocations.

Full text search

Reindexer has internal full text search engine. Full text search usage documentation and examples are here

Vector indexes (ANN/KNN)

Reindexer has internal k-nearest neighbors search engine. k-nearest neighbors search usage documentation and examples are here

Hybrid search

Reindexer has internal hybrid full text and k-nearest neighbors search engine. Its usage documentation and examples are here

Disk Storage

Reindexer can store documents to and load documents from disk via LevelDB. Documents are written to the storage backend asynchronously by large batches automatically in background.

When a namespace is created, all its documents are stored into RAM, so the queries on these documents run entirely in in-memory mode.

Replication

Reindexer supports synchronous and asynchronous replication. Check replication documentation here

Sharding

Reindexer has some basic support for sharding. Check sharding documentation here

Usage

Here is complete example of basic Reindexer usage:

package main

// Import package
import (
    "fmt"
    "math/rand"

    "github.com/restream/reindexer/v5"
    // choose how the Reindexer binds to the app (in this case "builtin," which means link Reindexer as a static library)
    _ "github.com/restream/reindexer/v5/bindings/builtin"

    // OR use Reindexer as standalone server and connect to it via TCP or unix domain socket (if available).
    // _ "github.com/restream/reindexer/v5/bindings/cproto"

    // OR link Reindexer as static library with bundled server.
    // _ "github.com/restream/reindexer/v5/bindings/builtinserver"
    // "github.com/restream/reindexer/v5/bindings/builtinserver/config"

)

// Define struct with reindex tags. Fields must be exported - private fields cannot be written into Reindexer
type Item struct {
    ID       int64  `reindex:"id,,pk"`    // 'id' is primary key
    Name     string `reindex:"name"`      // add index by 'name' field
    Articles []int  `reindex:"articles"`  // add index by articles 'articles' array
    Year     int    `reindex:"year,tree"` // add sortable index by 'year' field
}

func main() {
    // Init a database instance and choose the binding (builtin)
    db, err := reindexer.NewReindex("builtin:///tmp/reindex/testdb")

    // OR - Init a database instance and choose the binding (connect to server via TCP sockets)
    // Database should be created explicitly via reindexer_tool or via WithCreateDBIfMissing option:
    // If server security mode is enabled, then username and password are mandatory
    // db, err := reindexer.NewReindex("cproto://user:pass@127.0.0.1:6534/testdb", reindexer.WithCreateDBIfMissing())

    // OR - Init a database instance and choose the binding (connect to server via TCP sockets with TLS support
    // using cprotos-protocol and a package tls from the standard GO library)
    // Database should be created explicitly via reindexer_tool or via WithCreateDBIfMissing option:
    // If server security mode is enabled, then username and password are mandatory
    // It is assumed that reindexer RPC-server with TLS support is running at this address. 6535 is the default port for it.
    // tlsConfig := tls.Config{
    //  /*required options*/
    // }
    // db, err := reindexer.NewReindex("cprotos://user:pass@127.0.0.1:6535/testdb", reindexer.WithCreateDBIfMissing(), reindexer.WithTLSConfig(&tlsConfig))

    // OR - Init a database instance and choose the binding (connect to server via unix domain sockets)
    // Unix domain sockets are available on the unix systems only (socket file has to be explicitly set on the server's side with '--urpcaddr' option)
    // Database should be created explicitly via reindexer_tool or via WithCreateDBIfMissing option:
    // If server security mode is enabled, then username and password are mandatory
    // db, err := reindexer.NewReindex("ucproto://user:pass@/tmp/reindexer.socket:/testdb", reindexer.WithCreateDBIfMissing())

    // OR - Init a database instance and choose the binding (builtin, with bundled server)
    // serverConfig := config.DefaultServerConfig ()
    // If server security mode is enabled, then username and password are mandatory
    // db, err := reindexer.NewReindex("builtinserver://user:pass@testdb",reindexer.WithServerConfig(100*time.Second, serverConfig))

    // Check the error after rx instance init
    if err != nil {
        panic(err)
    }

    // Create new namespace with name 'items', which will store structs of type 'Item'
    db.OpenNamespace("items", reindexer.DefaultNamespaceOptions(), Item{})

    // Generate dataset
    for i := 0; i < 100000; i++ {
        err := db.Upsert("items", &Item{
            ID:       int64(i),
            Name:     "Vasya",
            Articles: []int{rand.Int() % 100, rand.Int() % 100},
            Year:     2000 + rand.Int()%50,
        })
        if err != nil {
            panic(err)
        }
    }

    // Query a single document
    elem, found := db.Query("items").
        Where("id", reindexer.EQ, 40).
        Get()

    if found {
        item := elem.(*Item)
        fmt.Println("Found document:", *item)
    }

    // Query multiple documents
    query := db.Query("items").
        Sort("year", false).                          // Sort results by 'year' field in ascending order
        WhereString("name", reindexer.EQ, "Vasya").   // 'name' must be 'Vasya'
        WhereInt("year", reindexer.GT, 2020).         // 'year' must be greater than 2020
        WhereInt("articles", reindexer.SET, 6, 1, 8). // 'articles' must contain one of [6,1,8]
        Limit(10).                                    // Return maximum 10 documents
        Offset(0).                                    // from 0 position
        ReqTotal()                                    // Calculate the total count of matching documents

    // Execute the query and return an iterator
    iterator := query.Exec()
    // Iterator must be closed
    defer iterator.Close()

    fmt.Println("Found", iterator.TotalCount(), "total documents, first", iterator.Count(), "documents:")

    // Iterate over results
    for iterator.Next() {
        // Get the next document and cast it to a pointer
        elem := iterator.Object().(*Item)
        fmt.Println(*elem)
    }
    // Check the error
    if err = iterator.Error(); err != nil {
        panic(err)
    }
}

There are also some basic samples for C++ and Go here

SQL compatible interface

As an alternative to the Query builder, Reindexer provides SQL-compatible query interface. Here is a sample of SQL interface usage:

```go ... iterator := db.ExecSQL ("SELECT * FROM items WHERE name='Vasya' AND year > 2020 AND articles IN (6,1,8) ORDER BY year