Kafka Streams Examples

[!NOTE] This repo is replaced with Confluent Tutorials for Apache Kafka. We still "keep the lights on", but we don't improve existing examples any longer, nor do we add new example.

This project contains code examples that demonstrate how to implement real-time applications and event-driven microservices using the Streams API of Apache Kafka aka Kafka Streams.

For more information take a look at the latest Confluent documentation on the Kafka Streams API, notably the Developer Guide

Table of Contents

• Available examples
  • Examples: Runnable Applications
  • Examples: Unit Tests
  • Examples: Integration Tests
  • Docker Example: Kafka Music demo application
  • Examples: Event Streaming Platform
• Requirements
  • Apache Kafka
  • Confluent Platform
  • Using IntelliJ or Eclipse
  • Java
  • Scala
• Packaging and running the examples
• Development
• Version Compatibility Matrix
• Where to find help

This repository has several branches to help you find the correct code examples for the version of Apache Kafka and/or Confluent Platform that you are using. See Version Compatibility Matrix below for details.

There are three kinds of examples:

• Examples under src/main/: These examples are short and concise. Also, you can interactively test-drive these examples, e.g. against a local Kafka cluster. If you want to actually run these examples, then you must first install and run Apache Kafka and friends, which we describe in section Packaging and running the examples. Each example also states its exact requirements and instructions at the very top.
• Examples under src/test/: These examples should test applications under src/main/. Unit Tests with TopologyTestDriver test the stream logic without external system dependencies. The integration tests use an embedded Kafka clusters, feed input data to them (using the standard Kafka producer client), process the data using Kafka Streams, and finally read and verify the output results (using the standard Kafka consumer client). These examples are also a good starting point to learn how to implement your own end-to-end integration tests.
• Ready-to-run Docker Examples: These examples are already built and containerized.

Additional examples may be found under src/main/.

See the documentation at Testing Streams Code.

We also provide several integration tests, which demonstrate end-to-end data pipelines. Here, we spawn embedded Kafka clusters and the Confluent Schema Registry, feed input data to them (using the standard Kafka producer client), process the data using Kafka Streams, and finally read and verify the output results (using the standard Kafka consumer client).

Additional examples may be found under src/test/.

Tip: Run mvn test to launch the tests.

***demonstrates how to probabilistically count items in an input stream by implementing a custom state store (CMSStore) that is backed by a Count-Min Sketch data structure (with the CMS implementation of Twitter Algebird)