Debezium Server

Debezium Server is a standalone Java application built on Quarkus framework. The application itself contains the core module and a set of modules responsible for communication with different target systems.

Building and Packaging

The following software is required to work with the Debezium codebase and build it locally:

• Git 2.2.1 or later
• JDK 11 or later, e.g. OpenJDK
• Docker Engine or Docker Desktop 1.9 or later
• Apache Maven 3.8.4 or later
  (or invoke the wrapper with ./mvnw for Maven commands)

See the links above for installation instructions on your platform. You can verify the versions are installed and running:

$ git --version
$ javac -version
$ mvn -version

Building the code

Debezium Server depends on core Debezium. You'll need to install the most recent snapshot locally. First obtain the code by cloning the Git repository:

$ git clone https://github.com/debezium/debezium.git
$ cd debezium

Then build the code using Maven:

$ mvn clean install -DskipITs -DskipTests

Then, you can build Debezium Server:

$ git clone https://github.com/debezium/debezium-server.git
$ cd debezium-server
$ mvn clean install -DskipITs -DskipTests

Creating a Distribution

Debezium Server is normally run by downloading the distribution tar.gz or zip file. You can generate this file by:

$ mvn clean package -DskipITs -DskipTests -Passembly

The archives can be found under debezium-server-dist/target.

Customizing Sink Modules in Distribution

By default, the distribution includes all available sink modules. You can customize which sinks are included using Maven profiles:

Include all sinks (default):

mvn clean package -DskipITs -DskipTests -Passembly

Create a custom distribution with specific sinks only:

First, use the custom-distribution profile to disable all sinks, then enable only the ones you need:

# Include only Kafka sink
mvn clean package -DskipITs -DskipTests -Passembly -Pcustom-distribution,sink-kafka

# Include multiple specific sinks (e.g., Kafka, Redis, and HTTP)
mvn clean package -DskipITs -DskipTests -Passembly -Pcustom-distribution,sink-kafka,sink-redis,sink-http

Available sink profiles: - sink-kinesis - Amazon Kinesis - sink-sqs - Amazon SQS - sink-sns - Amazon SNS - sink-http - HTTP - sink-pubsub - Google Cloud Pub/Sub - sink-pulsar - Apache Pulsar - sink-eventhubs - Azure Event Hubs - sink-redis - Redis - sink-kafka - Apache Kafka - sink-pravega - Pravega - sink-nats-streaming - NATS Streaming - sink-nats-jetstream - NATS JetStream - sink-infinispan - Infinispan - sink-rabbitmq - RabbitMQ - sink-rocketmq - Apache RocketMQ - sink-milvus - Milvus - sink-qdrant - Qdrant - sink-instructlab - InstructLab

This approach allows you to create smaller, more focused distributions that only include the sink modules you actually need.

Building just the artifacts, without running tests, CheckStyle, etc.

You can skip all non-essential plug-ins (tests, integration tests, CheckStyle, formatter, API compatibility check, etc.) using the "quick" build profile:

$ mvn clean verify -Dquick

This provides the fastest way for solely producing the output artifacts, without running any of the QA related Maven plug-ins. This comes in handy for producing connector JARs and/or archives as quickly as possible, e.g. for manual testing in Kafka Connect

Integration Tests

The per-module integration tests depend on the availability of the external services. It is thus recommended to execute integration tests per-module and set-up necessary pre-requisities beforehand.

Note: running these tests against external infrastructure may incur cost with your cloud provider. We're not going to pay your AWS/GCP/Azure bill.

Amazon Kinesis

• Execute aws configure as described in AWS CLI getting started guide and setup the account.
• Create Kinesis stream aws kinesis create-stream --stream-name testc.inventory.customers --shard-count 1
• Build the module and execute the tests mvn clean install -DskipITs=false -am -pl debezium-server-kinesis
• Remove the stream aws kinesis delete-stream --stream-name testc.inventory.customers

Google Cloud Pub/Sub

• Login into your Google Cloud account using gcloud auth application-default login as described in the documentation.
• Build the module and execute the tests mvn clean install -DskipITs=false -am -pl debezium-server-pubsub

Azure Event Hubs

Login into your Azure account and create a resource group, e.g. on the CLI:

az login
az group create --name eventhubstest --location westeurope

Create an Event Hubs namespace

Create an Event Hubs namespace. Check the documentation for options on how do this using the Azure Portal, Azure CLI etc., e.g. on the CLI:

az eventhubs namespace create --name debezium-test --resource-group eventhubstest -l westeurope

Create an Event Hub

Create an Event Hub (equivalent to a topic) with 5 partitions. Check the documentation for options on how do this using the Azure Portal, Azure CLI etc. , e.g. on the CLI:

`az eventhubs eventhub create` --name debezium-test-hub --resource-group eventhubstest --namespace-name debezium-test --partition-count 5

Build the module

Get the Connection string required to communicate with Event Hubs. The format is: Endpoint=sb://<NAMESPACE>/;SharedAccessKeyName=<ACCESS_KEY_NAME>;SharedAccessKey=<ACCESS_KEY_VALUE>. E.g. on the CLI:

az eventhubs namespace authorization-rule keys list --resource-group eventhubstest --namespace-name debezium-test --name RootManageSharedAccessKey

Set environment variables required for tests:

export EVENTHUBS_CONNECTION_STRING=<Event Hubs connection string>
export EVENTHUBS_NAME=<name of the Event hub created in previous step>

Execute the tests:

mvn clean install -DskipITs=false -Deventhubs.connection.string=$EVENTHUBS_CONNECTION_STRING -Deventhubs.hub.name=$EVENTHUBS_NAME -am -pl :debezium-server-eventhubs

Examine Events in the Event Hub

E.g. using kafkacat. Create kafkacat.conf:

metadata.broker.list=debezium-test.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanisms=PLAIN
sasl.username=$ConnectionString
sasl.password=Endpoint=sb://debezium-test.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<access key>

Start consuming events:

export KAFKACAT_CONFIG=

Clean up

Delete the Event Hubs namespace and log out, e.g. on the CLI:

az group delete -n eventhubstest
az logout