github.com/4paradigm/OpenMLDB @2.3.0

• Slack Channel
• Discussions
• 中文README

Introduction

FEDB is a NewSQL database optimised for online inferencing and decision making applications. These applications feed a pre-trained model with real-time features extracted from multiple time series windows for evaluating new data to support decision making. Existing in-memory databases cost hundreds or even thousands of milliseconds so they cannot meet the requirements of online inferencing and decisioning applications.

FEDB uses a double-layer skiplist as the core data structure. With all the data in memory and extreme compilation optimization of SQL, FEDB significantly reduces execution latency.

• High Performance

The benchmark shows that FEDB can be one to two orders of magnitude faster than SingleStore and SAP HANA.

• SQL Compatible

FEDB is compatible with most of ANSI SQL syntax. You can implement your applications with SQLAlchemy or JDBC.

• Online-offline Consistency

Machine learning applications powered by FEDB can be launched easily and ensure online and offline consistency, greatly reducing the cost of landing machine learning scenarios.

• High Availability

Support auto failover and scaling horizontally.

Note: The latest released FEDB is unstable and not recommend to be used in production environment.

Architecture

See more

Quick Start

Build on Linux

docker pull 4pdosc/centos6_gcc7_hybridsql:0.1.1
git clone https://github.com/4paradigm/fedb.git
cd fedb
docker run -v `pwd`:/fedb -it 4pdosc/centos6_gcc7_hybridsql:0.1.1
cd /fedb
sh steps/init_env.sh
mkdir -p build && cd build && cmake ../ && make -j5 fedb

Demo

• Predict taxi trip duration
• Detect the healthy of online transaction and make an alert -oncoming
• Online real-time transaction fraud detection -oncoming

Performance

In AI scenarios most real-time features are time-related and required to be computed over multiple time windows. So we use computation TopN queries as benchmark scenario.

Server Specification

The server spec is as follows:

Benchmark Results

The benchmark result shows that FEDB can be one to two orders of magnitude faster than SingleStore and SAP HANA. Please check our VLDB'21 paper for more benchmarks.

Roadmap

ANSI SQL Compatibility

FEDB is currently compatible with mainstream DDL and DML syntax, and will gradually enhances the compatibility of ANSI SQL syntax.

• [2021H1] Support the standard syntax of Window, Where, Group By and Join ect.
• [2021H1&H2] Expand AI-oriented SQL syntax and UDAF functions.

Features

In order to meet the high performance requirements of realtime inference and decisioning scenarios, FEDB chooses memory as the storage engine medium. At present, the memory storage engine used in the industry has memory fragmentation and recovery efficiency problems. FEDB plans to optimize the memory allocation algorithm to reduce fragmentation and accelerate data recovery with PMEM(Intel Optane DC Persistent Memory Module).

• [2021H1]Provide a new strategy of memory allocation to reduce memory fragmentation.
• [2021H2]Support PMEM-based storage engine.

Build Ecosystem

FEDB has python client and java client which support most of JDBC API. FEDB will make a connection with big data ecosystem for integrating with Flink/Kafka/Spark simplily.

• [2021H1&H2]Support Flink/Kafka/Spark connector.

Feedback and Getting involved

• Report bugs, ask questions or give suggestions by Github Issues.
• Cannot find what you are looking for? Have a question or idea? Please post your questions or comments on our slack.

License

Apache License 2.0