BuntDB is a low-level, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a single writer. It supports custom indexes and geospatial data. It's ideal for projects that need a dependable database and favor speed over data size.

Features

• In-memory database for fast reads and writes
• Embeddable with a simple API
• Spatial indexing for up to 20 dimensions; Useful for Geospatial data
• Index fields inside JSON documents
• Collate i18n Indexes using the optional collate package
• Create custom indexes for any data type
• Support for multi value indexes; Similar to a SQL multi column index
• Built-in types that are easy to get up & running; String, Uint, Int, Float
• Flexible iteration of data; ascending, descending, and ranges
• Durable append-only file format for persistence
• Option to evict old items with an expiration TTL
• ACID semantics with locking transactions that support rollbacks

Getting Started

Installing

To start using BuntDB, install Go and run go get:

$ go get -u github.com/tidwall/buntdb

This will retrieve the library.

Opening a database

The primary object in BuntDB is a DB. To open or create your database, use the buntdb.Open() function:

package main

import (
    "log"

    "github.com/tidwall/buntdb"
)

func main() {
    // Open the data.db file. It will be created if it doesn't exist.
    db, err := buntdb.Open("data.db")
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    ...
}

It's also possible to open a database that does not persist to disk by using :memory: as the path of the file.

buntdb.Open(":memory:") // Open a file that does not persist to disk.

Transactions

All reads and writes must be performed from inside a transaction. BuntDB can have one write transaction opened at a time, but can have many concurrent read transactions. Each transaction maintains a stable view of the database. In other words, once a transaction has begun, the data for that transaction cannot be changed by other transactions.

Transactions run in a function that exposes a Tx object, which represents the transaction state. While inside a transaction, all database operations should be performed using this object. You should never access the origin DB object while inside a transaction. Doing so may have side-effects, such as blocking your application.

When a transaction fails, it will roll back, and revert all changes that occurred to the database during that transaction. There's a single return value that you can use to close the transaction. For read/write transactions, returning an error this way will force the transaction to roll back. When a read/write transaction succeeds all changes are persisted to disk.

Read-only Transactions

A read-only transaction should be used when you don't need to make changes to the data. The advantage of a read-only transaction is that there can be many running concurrently.

err := db.View(func(tx *buntdb.Tx) error {
    ...
    return nil
})

Read/write Transactions

A read/write transaction is used when you need to make changes to your data. There can only be one read/write transaction running at a time. So make sure you close it as soon as you are done with it.

err := db.Update(func(tx *buntdb.Tx) error {
    ...
    return nil
})

Setting and getting key/values

To set a value you must open a read/write transaction:

err := db.Update(func(tx *buntdb.Tx) error {
    _, _, err := tx.Set("mykey", "myvalue", nil)
    return err
})

To get the value:

err := db.View(func(tx *buntdb.Tx) error {
    val, err := tx.Get("mykey")
    if err != nil{
        return err
    }
    fmt.Printf("value is %s\n", val)
    return nil
})

Getting non-existent values will cause an ErrNotFound error.

Iterating

All keys/value pairs are ordered in the database by the key. To iterate over the keys:

err := db.View(func(tx *buntdb.Tx) error {
    err := tx.Ascend("", func(key, value string) bool {
        fmt.Printf("key: %s, value: %s\n", key, value)
        return true // continue iteration
    })
    return err
})

There is also AscendGreaterOrEqual, AscendLessThan, AscendRange, AscendEqual, Descend, DescendLessOrEqual, DescendGreaterThan, DescendRange, and DescendEqual. Please see the documentation for more information on these functions.

Custom Indexes

Initially all data is stored in a single B-tree with each item having one key and one value. All of these items are ordered by the key. This is great for quickly getting a value from a key or iterating over the keys. Feel free to peruse the B-tree implementation.

You can also create custom indexes that allow for ordering and iterating over values. A custom index also uses a B-tree, but it's more flexible because it allows for custom ordering.

For example, let's say you want to create an index for ordering names:

db.CreateIndex("names", "*", buntdb.IndexString)

This will create an index named names which stores and sorts all values. The second parameter is a pattern that is used to filter on keys. A * wildcard argument means that we want to accept all keys. IndexString is a built-in function that performs case-insensitive ordering on the values

Now you can add various names:

db.Update(func(tx *buntdb.Tx) error {
    tx.Set("user:0:name", "tom", nil)
    tx.Set("user:1:name", "Randi", nil)
    tx.Set("user:2:name", "jane", nil)
    tx.Set("user:4:name", "Janet", nil)
    tx.Set("user:5:name", "Paula", nil)
    tx.Set("user:6:name", "peter", nil)
    tx.Set("user:7:name", "Terri", nil)
    return nil
})

Finally you can iterate over the index:

db.View(func(tx *buntdb.Tx) error {
    tx.Ascend("names", func(key, val string) bool {
    fmt.Printf(buf, "%s %s\n", key, val)
        return true
    })
    return nil
})

The output should be:

user:2:name jane
user:4:name Janet
user:5:name Paula
user:6:name peter
user:1:name Randi
user:7:name Terri
user:0:name tom

The pattern parameter can be used to filter on keys like this:

db.CreateIndex("names", "user:*", buntdb.IndexString)

Now only items with keys that have the prefix user: will be added to the names index.

Built-in types

Along with IndexString, there is also IndexInt, IndexUint, and IndexFloat. These are built-in types for indexing. You can choose to use these or create your own.

So to create an index that is numerically ordered on an age key, we could use:

db.CreateIndex("ages", "user:*:age", buntdb.IndexInt)

And then add values:

db.Update(func(tx *buntdb.Tx) error {
    tx.Set("user:0:age", "35", nil)
    tx.Set("user:1:age", "49", nil)
    tx.Set("user:2:age", "13", nil)
    tx.Set("user:4:age", "63", nil)
    tx.Set("user:5:age", "8", nil)
    tx.Set("user:6:age", "3", nil)
    tx.Set("user:7:age", "16", nil)
    return nil
})

db.View(func(tx *buntdb.Tx) error {
    tx.Ascend("ages", func(key, val string) bool {
    fmt.Printf(buf, "%s %s\n", key, val)
        return true
    })
    return nil
})

The output should be:

user:6:age 3
user:5:age 8
user:2:age 13
user:7:age 16
user:0:age 35
user:1:age 49
user:4:age 63

Spatial Indexes

BuntDB has support for spatial indexes by storing rectangles in an R-tree. An R-tree is organized in a similar manner as a B-tree, and both are balanced trees. But, an R-tree is special because it can operate on data that is in multiple dimensions. This is super handy for Geospatial applications.

To create a spatial index use the CreateSpatialIndex function:

db.CreateSpatialIndex("fleet", "fleet:*:pos", buntdb.IndexRect)

Then IndexRect is a built-in function that converts rect strings to a format that the R-tree can use. It's easy to use this function out of the box, but you might find it better to create a custom one that renders from a different format, such as Well-known text or GeoJSON.

To add some lon,lat points to the fleet index:

db.Update(func(tx *buntdb.Tx) error {
    tx.Set("fleet:0:pos", "[-115.567 33.532]", nil)
    tx.Set("fleet:1:pos", "[-116.671 35.735]", nil)
    tx.Set("fleet:2:pos", "[-113.902 31.234]", nil)
    return nil
})

And then you can run the Intersects function on the index:

db.View(func(tx *buntdb.Tx) error {
    tx.Intersects("fleet", "[-117 30],[-112 36]", func(key, val string) bool {
        ...
        return true
    })
    return nil
})

This will get all three positions.

k-Nearest Neighbors

Use the Nearby function to get all the positions in order of nearest to farthest :

db.View(func(tx *buntdb.Tx) error {
    tx.Nearby("fleet", "[-113 33]", func(key, val string, dist float64) bool {
        ...
        return true
    })
    return nil
})

Spatial bracket syntax

The bracket syntax [-117 30],[-112 36] is unique to BuntDB, and it's how the built-in rectangles are processed. But, you are not limited to this syntax. Whatever Rect function you choose to use during CreateSpatialIndex will be used to process the parameter, in this case it's IndexRect.

• 2D rectangle: [10 15],[20 25] Min XY: "10x15", Max XY: "20x25"

• 3D rectangle: [10 15 12],[20 25 18] Min XYZ: "10x15x12", Max XYZ: "20x25x18"

• 2D point: [10 15] XY: "10x15"

• LonLat point: [-112.2693 33.5123] LatLon: "33.5123 -112.2693"

• LonLat bounding box: [-112.26 33.51],[-112.18 33.67] Min LatLon: "33.51 -112.26", Max LatLon: "33.67 -112.18"

Notice: The longitude is the Y axis and is on the left, and latitude is the X axis and is on the right.

You can also represent Infinity by using -inf and +inf. For example, you might have the following points ([X Y M] where XY is a point and M is a timestamp):

[3 9 1]
[3 8 2]
[4 8 3]
[4 7 4]
[5 7 5]
[5 6 6]

You can then do a search for all points with M between 2-4 by calling Intersects.

tx.Intersects("points", "[-inf -inf 2],[+inf +inf 4]", func(key, val string) bool {
    println(val)
    return true
})

Which will return:

[3 8 2]
[4 8 3]
[4 7 4]

JSON Indexes

Indexes can be created on individual fields inside JSON documents. BuntDB uses GJSON under the hood.

For example:

package main

import (
    "fmt"

    "github.com/tidwall/buntdb"
)

func main() {
    db, _ := buntdb.Open(":memory:")
    db.CreateIndex("last_name", "*", buntdb.IndexJSON("name.last"))
    db.CreateIndex("age", "*", buntdb.IndexJSON("age"))
    db.Update(func(tx *buntdb.Tx) error {
        tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":38}`, nil)
        tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":47}`, nil)
        tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":52}`, nil)
        tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":28}`, nil)
        return nil
    })
    db.View(func(tx *buntdb.Tx) error {
        fmt.Println("Order by last name")
        tx.Ascend("last_name", func(key, value string) bool {
            fmt.Printf("%s: %s\n", key, value)
            return true
        })
        fmt.Println("Order by age")
        tx.Ascend("age", func(key, value string) bool {
            fmt.Printf("%s: %s\n", key, value)
            return true
        })
        fmt.Println("Order by age range 30-50")
        tx.AscendRange("age", `{"age":30}`, `{"age":50}`, func(key, value string) bool {
            fmt.Printf("%s: %s\n", key, value)
            return true
        })
        return nil
    })
}

Results:

Order by last name
3: {"name":{"first":"Carol","last":"Anderson"},"age":52}
4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
2: {"name":{"first":"Janet","last":"Prichard"},"age":47}

Order by age
4: {"name":{"first":"Alan","last":"Cooper"},"age":28}
1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
2: {"name":{"first":"Janet","last":"Prichard"},"age":47}
3: {"name":{"first":"Carol","last":"Anderson"},"age":52}

Order by age range 30-50
1: {"name":{"first":"Tom","last":"Johnson"},"age":38}
2: {"name":{"first":"Janet","last":"Prichard"},"age":47}

Multi Value Index

With BuntDB it's possible to join multiple values on a single index. This is similar to a multi column index in a traditional SQL database.

In this example we are creating a multi value index on "name.last" and "age":

``go db, _ := buntdb.Open(":memory:") db.CreateIndex("last_name_age", "*", buntdb.IndexJSON("name.last"), buntdb.IndexJSON("age")) db.Update(func(tx *buntdb.Tx) error { tx.Set("1",{"name":{"first":"Tom","last":"Johnson"},"age":38}, nil) tx.Set("2",{"name":{"first":"Janet","last":"Prichard"},"age":47}, nil) tx.Set("3",{"name":{"first":"Carol","last":"Anderson"},"age":52}, nil) tx.Set("4",{"name":{"first":"Alan","last":"Cooper"},"age":28}, nil) tx.Set("5",{"name":{"first":"Sam","last":"Anderson"},"age":51}, nil) tx.Set("6",{"name":{"first":"Melinda","last":"Prichard"},"age":44}`, nil) return nil }) db.View(func(tx *buntdb.Tx) error { tx.Ascend("last_name_age", func(key, v