English | 中文

Introduction

• ape-dts is a data migration tool enabling any-to-any data transfers.
• It also provides data subscription and data processing.
• It is lightweight, efficient and standalone, requiring no third-party components or extra storage.
• Designed for cloud-native stateless component scenarios.
• In Rust.

Key features

• Supports data migration between various databases, both homogeneous and heterogeneous.
• Supports snapshot and cdc tasks with resume from breakpoint.
• Supports checking and revising data.
• Supports filtering and routing at the database, table, and column levels.
• Implements different parallel algorithms for different sources, targets, and task types to improve performance.
• Allows loading user-defined Lua scripts to modify the data.

Supported task types

Advanced

Crate features

The dt-main crate provides several optional components which can be enabled via Cargo [features]:

• metrics: Enable Prometheus format task metrics HTTP service interface. After enabling this feature, you can customize the metrics service with the following configuration:

[metrics] # http service host http_host=127.0.0.1 # http service port http_port=9090 # http service worker count workers=2 # prometheus metrics const labels labels=your_label1:your_value1,your_label2:your_value2

• TBD

Quick starts

Tutorial

• prerequisites
• mysql -> mysql
• pg -> pg
• mongo -> mongo
• redis -> redis
• mysql -> starrocks
• mysql -> doris
• mysql -> clickhouse
• mysql -> tidb
• mysql -> kafka -> consumer
• pg -> starrocks
• pg -> doris
• pg -> clickhouse
• pg -> kafka -> consumer
• snapshot + cdc without data loss
• modify data by lua

Run tests

Refer to test docs for details.

More docs

• Configurations
• config details
• Structure tasks
• migration
• check
• check by Liquibase
• Snapshot tasks
• data migration
• data check
• data revise
• data review
• resume at breakpoint
• multiple tables in parallel
• CDC tasks
• data sync
• heartbeat to source database
• two-way data sync
• generate sqls from CDC
• resume at breakpoint
• Custom consumers
• mysql/pg -> kafka -> consumer
• Data processing
• modify data by lua
• Monitor
• monitor info
• position info
• Task templates
• mysql -> mysql
• pg -> pg
• mongo -> mongo
• redis -> redis
• mysql/pg -> kafka
• mysql -> starrocks
• mysql -> doris
• mysql -> clickhouse
• pg -> starrocks
• pg -> doris
• pg -> clickhouse

Benchmark

• MySQL -> MySQL, Snapshot

• MySQL -> MySQL, CDC

• Image size

• more benchmark details

Contributing

Structure

Modules

• dt-main: program entry
• dt-precheck: pre-check, to minimize interruptions during subsequent data operations by identifying issues early for fast failure
• dt-connector: extractors + sinkers for databases
• dt-pipeline: pipeline to connect extractors and sinkers
• dt-parallelizer: parallel algorithms
• dt-task: create extractors + sinkers + pipelines + parallelizers according to configurations
• dt-common: common utils, basic data structures, metadata management

• dt-tests: integration tests

• related sub module: mysql binlog connector in rust

Build

• Minimum supported Rust version (MSRV) The current minimum supported Rust version (MSRV) is 1.85.0.
• cargo build
• build images

Checklist

• run cargo clippy --all-targets --all-features --workspace fix all clippy issues.

Community

If you have any questions, you can reach out to us through:

• ApeDTS GitHub Discussions
• ApeDTS Wechat Account with note ape-dts: