EJDB 2.0

EJDB2 is an embeddable JSON database engine published under MIT license.

The Story of the IT-depression, birds and EJDB 2.0

• C11 API
• Single file database
• Online backups support
• 500K library size for Android
• iOS / Android / React Native / Flutter integration
• Simple but powerful query language (JQL) as well as support of the following standards:
• rfc6902 JSON Patch
• rfc7386 JSON Merge patch
• rfc6901 JSON Path
• Support of collection joins
• Powered by iowow.io - The persistent key/value storage engine
• HTTP REST/Websockets endpoints powered by IWNET and BearSSL.
• JSON documents are stored in using fast and compact binn binary format

---

• Native language bindings
• Supported platforms
• macOS
• iOS
• Linux
• Android
• Windows
• JQL query language
• Grammar
• Quick into
• Data modification
• Projections
• Collection joins
• Sorting
• Query options
• Indexes and performance
• Network API
• HTTP API
• Websockets API
• C API
• License

---

EJDB2 platforms matrix

	Linux	macOS	iOS	Android	Windows
C library	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:1
NodeJS	:heavy_check_mark:	:heavy_check_mark:			:x:3
DartVM	:heavy_check_mark:	:heavy_check_mark:2			:x:3
Flutter			:heavy_check_mark:	:heavy_check_mark:
React Native			:x:4	:heavy_check_mark:
Swift	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
Java	:heavy_check_mark:	:heavy_check_mark:		:heavy_check_mark:	:heavy_check_mark:2

[1] No HTTP/Websocket support #257

[2] Binaries are not distributed with dart pub. You can build it manually

[3] Can be build, but needed a linkage with windows node/dart libs.

[4] Porting in progress #273

Native language bindings

• NodeJS
• Dart
• Java
• Android support
• Swift | iOS
• React Native
• Flutter

Unofficial EJDB2 language bindings

• Go
• https://github.com/memmaker/go-ejdb2
• Rust
• https://crates.io/crates/ejdb2
• .Net
• https://github.com/kmvi/ejdb2-csharp
• Haskell
• https://github.com/cescobaz/ejdb2haskell
• https://hackage.haskell.org/package/ejdb2-binding
• Pharo
• https://github.com/pharo-nosql/pharo-ejdb
• Lua
• https://github.com/chriku/ejdb-lua

Status

• EJDB 2.0 core engine is well tested and used in various heavily loaded deployments
• Tested on Linux, macOS and FreeBSD. Has limited Windows support
• Old EJDB 1.x version can be found in separate ejdb_1.x branch. We are not maintaining ejdb 1.x.

Used by

• Wirow video conferencing platform

Are you using EJDB? Let me know!

macOS

EJDB2 code ported and tested on High Sierra / Mojave / Catalina

EJDB2 Swift binding for MacOS, iOS and Linux. Swift binding is outdated at now. Looking for contributors.

brew install ejdb

Building from sources

cmake v3.12 or higher required

git clone --recurse-submodules git@github.com:Softmotions/ejdb.git

mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make install

Linux

Building debian packages

mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DPACKAGE_DEB=ON
make package

RPM based Linux distributions

mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DPACKAGE_RPM=ON
make package

Windows

EJDB2 can be cross-compiled for windows

Note: HTTP/Websocket network API is disabled and not yet supported

Nodejs/Dart bindings not yet ported to Windows.

Cross-compilation Guide for Windows

IWSTART

IWSTART is an automatic CMake initial project generator for C projects based on iowow / iwnet / ejdb2 libs.

https://github.com/Softmotions/iwstart

Android

• Flutter binding
• React Native binding

Sample Android application

• https://github.com/Softmotions/ejdb/tree/master/src/bindings/ejdb2_android/test

• https://github.com/Softmotions/ejdb_android_todo_app

JQL

EJDB query language (JQL) syntax inspired by ideas behind XPath and Unix shell pipes. It designed for easy querying and updating sets of JSON documents.

JQL grammar

JQL parser created created by peg/leg — recursive-descent parser generators for C Here is the formal parser grammar: https://github.com/Softmotions/ejdb/blob/master/src/jql/jqp.leg

Non formal JQL grammar adapted for brief overview

Notation used below is based on SQL syntax description:

Rule	Description
' '	String in single quotes denotes unquoted string literal as part of query.
{ a | b }	Curly brackets enclose two or more required alternative choices, separated by vertical bars.
[ ]	Square brackets indicate an optional element or clause. Multiple elements or clauses are separated by vertical bars.
|	Vertical bars separate two or more alternative syntax elements.
...	Ellipses indicate that the preceding element can be repeated. The repetition is unlimited unless otherwise indicated.
( )	Parentheses are grouping symbols.
Unquoted word in lower case	Denotes semantic of some query part. For example: placeholder_name - name of any placeholder.

QUERY = FILTERS [ '|' APPLY ] [ '|' PROJECTIONS ] [ '|' OPTS ];

STR = { quoted_string | unquoted_string };

JSONVAL = json_value;

PLACEHOLDER = { ':'placeholder_name | '?' }

FILTERS = FILTER [{ and | or } [ not ] FILTER];

  FILTER = [@collection_name]/NODE[/NODE]...;

  NODE = { '*' | '**' | NODE_EXPRESSION | STR };

  NODE_EXPRESSION = '[' NODE_EXPR_LEFT OP NODE_EXPR_RIGHT ']'
                        [{ and | or } [ not ] NODE_EXPRESSION]...;

  OP =   [ '!' ] { '=' | '>=' | '<=' | '>' | '<' | ~ }
      | [ '!' ] { 'eq' | 'gte' | 'lte' | 'gt' | 'lt' }
      | [ not ] { 'in' | 'ni' | 're' };

  NODE_EXPR_LEFT = { '*' | '**' | STR | NODE_KEY_EXPR };

  NODE_KEY_EXPR = '[' '*' OP NODE_EXPR_RIGHT ']'

  NODE_EXPR_RIGHT =  JSONVAL | STR | PLACEHOLDER

APPLY = { 'apply' | 'upsert' } { PLACEHOLDER | json_object | json_array  } | 'del'

OPTS = { 'skip' n | 'limit' n | 'count' | 'noidx' | 'inverse' | ORDERBY }...

  ORDERBY = { 'asc' | 'desc' } PLACEHOLDER | json_path

PROJECTIONS = PROJECTION [ {'+' | '-'} PROJECTION ]

  PROJECTION = 'all' | json_path

• json_value: Any valid JSON value: object, array, string, bool, number.
• json_path: Simplified JSON pointer. Eg.: /foo/bar or /foo/"bar with spaces"/
• * in context of NODE: Any JSON object key name at particular nesting level.
• ** in context of NODE: Any JSON object key name at arbitrary nesting level.
• * in context of NODE_EXPR_LEFT: Key name at specific level.
• ** in context of NODE_EXPR_LEFT: Nested array value of array element under specific key.

JQL quick introduction

Lets play with some very basic data and queries. For simplicity we will use ejdb websocket network API which provides us a kind of interactive CLI. The same job can be done using pure C API too (ejdb2.h jql.h).

NOTE: Take a look into JQL test cases for more examples.

{
  "firstName": "John",
  "lastName": "Doe",
  "age": 28,
  "pets": [
    {"name": "Rexy rex", "kind": "dog", "likes": ["bones", "jumping", "toys"]},
    {"name": "Grenny", "kind": "parrot", "likes": ["green color", "night", "toys"]}
  ]
}

Save json as sample.json then upload it the family collection:

# Start HTTP/WS server protected by some access token
./jbs -a 'myaccess01'
8 Mar 16:15:58.601 INFO: HTTP/WS endpoint at localhost:9191

Server can be accessed using HTTP or Websocket endpoint. More info

curl -d '@sample.json' -H'X-Access-Token:myaccess01' -X POST http://localhost:9191/family

We can play around using interactive wscat websocket client.

wscat  -H 'X-Access-Token:myaccess01' -c http://localhost:9191
connected (press CTRL+C to quit)
> k info
< k     {
 "version": "2.0.0",
 "file": "db.jb",
 "size": 8192,
 "collections": [
  {
   "name": "family",
   "dbid": 3,
   "rnum": 1,
   "indexes": []
  }
 ]
}

> k get family 1
< k     1       {
 "firstName": "John",
 "lastName": "Doe",
 "age": 28,
 "pets": [
  {
   "name": "Rexy rex",
   "kind": "dog",
   "likes": [
    "bones",
    "jumping",
    "toys"
   ]
  },
  {
   "name": "Grenny",
   "kind": "parrot",
   "likes": [
    "green color",
    "night",
    "toys"
   ]
  }
 ]
}

Note about the k prefix before every command; It is an arbitrary key chosen by client and designated to identify particular websocket request, this key will be returned with response to request and allows client to identify that response for his particular request. More info

Query command over websocket has the following format:

<key> query <collection> <query>

So we will consider only <query> part in this document.

Get all elements in collection

k query family /*

or

k query family /**

or specify collection name in query explicitly

k @family/*

We can execute query by HTTP POST request

curl --data-raw '@family/[firstName = John]' -H'X-Access-Token:myaccess01' -X POST http://localhost:9191

1   {"firstName":"John","lastName":"Doe","age":28,"pets":[{"name":"Rexy rex","kind":"dog","likes":["bones","jumping","toys"]},{"name":"Grenny","kind":"parrot","likes":["green color","night","toys"]}]}

Set the maximum number of elements in result set

k @family/* | limit 10

Get documents where specified json path exists

Element at index 1 exists in likes array within a pets sub-object

> k query family /pets/*/likes/1
< k     1       {"firstName":"John"...

Element at index 1 exists in likes array at any likes nesting level

> k query family /**/likes/1
< k     1       {"firstName":"John"...

From this point and below I will omit websocket specific prefix k query family and consider only JQL queries.

Get documents by primary key

In order to get documents by primary key the following options are available:

1. Use API call ejdb_get() ts const doc = await db.get('users', 112);

2. Use the special query construction: /=:? or @collection/=:?

Get document from users collection with primary key 112

> k @users/=112

Update tags array for document in jobs collection (TypeScript):

 await db.createQuery('@jobs/ = :? | apply :? | count')
    .setNumber(0, id)
    .setJSON(1, { tags })
    .completionPromise();

Array of primary keys can also be used for matching:

```ts await db.createQuery(