github.com/ankur-anand/unisondb @main

UnisonDB

Store, stream, and sync instantly — UnisonDB is a log-native, real-time database that replicates like a message bus for AI and Edge Computing.

What UnisonDB Is

UnisonDB is an open-source database designed specifically for Edge AI and Edge Computing.

It is a reactive, log-native and multi-model database built for real-time and edge-scale applications. UnisonDB combines a B+Tree storage engine with WAL-based (Write-Ahead Logging) replication over gRPC or object-store backed replication, enabling fan-out replication across many readers while preserving strong consistency and durability.

Replication Model

Writes are committed by a Raft quorum on the write servers (if enabled); read-only edge replicas and relayers can consume WAL through either a live gRPC stream or blob-backed replication using object storage.

Blob-backed replication changes the fan-out model:

• The writer publishes WAL as immutable segment files plus bounded catalog metadata
• Any number of readers can poll and catch up directly from object storage
• Teams already running S3, MinIO, GCS, or Azure Blob do not need an always-on gRPC replication path for every replica

See cmd/examples/blobstore-minio for a local MinIO example using the same S3-compatible provider path.

Key Features

• High Availability Writes: Raft consensus on write servers (quorum acks); relayer/replica use in-sync replica (ISR) replication
• Streaming Replication: WAL replication over gRPC or object-store backed replication
• Blob Fan-Out: Publish WAL once into object storage and let N readers poll directly from S3/MinIO/GCS/Azure-backed replication stores
• Multi-Modal Storage: Key-Value, Wide-Column, and Large Objects (LOB)
• Real-Time Notifications: ZeroMQ-based(Side-car) change notifications with sub-millisecond latency
• Durable: B+Tree storage with Write-Ahead Logging
• Edge-First Design: Optimized for edge computing and local-first architectures
• Namespace Isolation: Multi-tenancy support with namespace-based isolation

Use Cases

UnisonDB is built for distributed edge-first architectures systems where data and computation must live close together — reducing network hops, minimizing latency, and enabling real-time responsiveness at scale.

By co-locating data with the services that use it, UnisonDB removes the traditional boundary between the database and the application layer. Applications can react to local changes instantly, while UnisonDB’s WAL-based replication ensures eventual consistency across all replicas globally.

Fan-Out Scaling

UnisonDB can fan out updates to 100+ edge nodes in just a few milliseconds from a single upstream—and because it supports multi-hop relaying, that reach compounds naturally. Each hop carries the network + application latency of its link;

In a simple 2-hop topology: - Hop 1: Primary → 100 hubs (≈250–500ms) - Hop 2: Each hub → 100 downstream edge nodes (similar latency) - Total reach: 100 + 10,000 = 10,100 nodes

Even at 60k–80k SET ops/sec with 1 KB values, UnisonDB can propagate those updates across 10,000+ nodes within seconds—without Kafka, Pub/Sub, CDC pipelines, or heavyweight brokers. (See the Relayer vs Latency benchmarks below for measured numbers.)

Quick Start

# Clone the repository
git clone https://github.com/ankur-anand/unisondb
cd unisondb

# Build
go build -o unisondb ./cmd/unisondb

# Run in server mode (primary)
./unisondb server --config config.toml

# Use the HTTP API
curl -X PUT http://localhost:4000/api/v1/default/kv/mykey \
  -H "Content-Type: application/json" \
  -d '{"value":"bXl2YWx1ZQ=="}'

Documentation

1. Getting Started with UnisonDB
2. Complete Configuration Guide
3. Architecture Overview
4. HTTP API Reference
5. Backup and Restore
6. Deployment Topologies
7. Rough Roadmap

UnisonDB implements a pluggable storage backend architecture supporting two BTree implementations: - BoltDB: Single-file, ACID-compliant BTree. - LMDB: Memory-mapped ACID-compliant BTree with copy-on-write semantics.

Redis-Compatible Benchmark: UnisonDB vs BadgerDB vs BoltDB vs LMDB

This benchmark compares the write and read performance of four databases — UnisonDB, BadgerDB, LMDB and BoltDB* — using a Redis-compatible interface and the official redis-benchmark tool.

What We Measured

• Throughput: Requests per second for SET (write) and GET (read) operations
• Latency: p50 latency in milliseconds
• Workload: 50 iterations of mixed SET and GET operations (200k ops per run)
• Concurrency: 10 parallel clients, 10 pipelined requests, 4 threads
• Payload Size: 1KB

Test Environment

Chip: Apple M2 Pro
Total Number of Cores: 10 (6 performance and 4 efficiency)
Memory: 16 GB
Unisondb Btree Backend - LMDB

All three databases were tested under identical conditions to highlight differences in write path efficiency, read performance, and I/O characteristics. The Redis-compatible server implementation can be found in internal/benchtests/cmd/redis-server/.

Results

Performance Testing: Local Replication

Test Setup

We validated the WAL-based replication architecture using the pkg/replicator component. It Uses the same redis-compatible bench tool but this time the server is started a n=[100,200,500,750,1000] goroutine that is an independent WAL reader, capturing critical performance metrics:

1. Physical Latency Tracking: Measures p50, p90, p99, and max latencies Vs Relayer.
2. SET, GET Latency vs Relayer
3. SET, GET Throughput Vs Relayer.

Results

Test Replication Flow

Why UnisonDB

Traditional databases persist. Stream systems propagate. UnisonDB does both — turning every write into a durable, queryable stream that replicates seamlessly across the edge.

The Problem: Storage and Streaming Live in Different Worlds

Modern systems are reactive — every change needs to propagate instantly to dashboards, APIs, caches, and edge devices.
Yet, databases were built for persistence, not propagation.

You write to a database, then stream through Kafka.
You replicate via CDC.
You patch syncs between cache and storage.

This split between state and stream creates friction: - Two systems to maintain and monitor - Eventual consistency between write path and read path - Network latency on every read or update - Complex fan-out when scaling to hundreds of edges

The Gap

LMDB and BoltDB excel at local speed — but stop at one node.
etcd and Consul replicate state — but are consensus-bound and small-cluster only.
Kafka and NATS stream messages — but aren’t queryable databases.

The Solution: Log-Native by Design

UnisonDB fuses database semantics with streaming mechanics — the log is the database.
Every write is durable, ordered, and instantly available as a replication stream.

No CDC, no brokers, no external pipelines.
Just one unified engine that:

• Stores data in B+Trees for predictable reads
• Streams data via WAL replication to thousands of nodes
• Reacts instantly with sub-second fan-out
• Keeps local replicas fully queryable, even offline

UnisonDB eliminates the divide between “database” and “message bus,”
enabling reactive, distributed, and local-first systems — without the operational sprawl.

UnisonDB collapses two worlds — storage and streaming — into one unified log-native core.
The result: a single system that stores, replicates, and reacts — instantly.

Core Architecture

UnisonDB is built on three foundational layers:

1. WALFS - Write-Ahead Log File System (mmap-based, optimized for reading at scale).
2. Engine - Hybrid storage combining WAL, MemTable, and B-Tree
3. Replication - WAL-based streaming with offset tracking

The Layered View

UnisonDB stacks a multi-model engine on top of WALFS — a log-native core that unifies storage, replication, and streaming into one continuous data flow.

+-----------------------------------------------------------+
|                Multi-Model API Layer                      |
|       (KV, Wide-Column, LOB, Txn Engine, Query)           |
+-----------------------------------------------------------+
|                   Engine Layer                            |
|   WALFS-backed MemTable + B-Tree Store                    |
|   (writes → WALFS, reads → B-Tree + MemTable)             |
+-----------------------------------------------------------+
|          WALFS (Core Log)          |  Replication Layer   |
|  Append-only, mmap-based           |  WAL-based streaming |
|  segmented log                     |  (followers tail WAL)|
|  Commit-ordered, replication-safe  |  Offset tracking,    |
|                                    |  catch-up, tailing   |
+-----------------------------------------------------------+
|                       Disk                                |
+-----------------------------------------------------------+

1. WALFS (Write-Ahead Log)

Overview

WALFS is a memory-mapped, segmented write-ahead log implementation designed for both writing AND reading at scale. Unlike traditional WALs that optimize only for sequential writes, WALFS provides efficient random access for replication, and real-time tailing.

Segment Structure

Each WALFS segment consists of two regions:

+----------------------+-----------------------------+-------------+
|   Segment Header     |        Record 1             |  Record 2   |
|     (64 bytes)       |  Header + Data + Trailer    |     ...     |
+----------------------+-----------------------------+-------------+

Segment Header (64 bytes)

Record Format (8-byte aligned)

Each record is written in its own aligned frame:

WALFS Reader Capabilities

WALFS provides powerful reading capabilities essential for replication and recovery:

1. Forward-Only Iterator

```go reader := walLog.NewReader() defer reader.Close()

for { data, pos, err := reader.Next() if err == io.EOF { break }