hub / github.com/aws-samples/amazon-managed-service-for-apache-flink-examples

github.com/aws-samples/amazon-managed-service-for-apache-flink-examples @main

680 symbols 1,383 edges 135 files 107 documented · 16%

What it actually does AI analysis from the code graph — generated when you open this

loading…

README

Amazon Managed Service for Apache Flink Examples

Example applications in Java, Python, Scala and SQL for Amazon Managed Service for Apache Flink (formerly known as Amazon Kinesis Data Analytics), illustrating various aspects of Apache Flink applications, and simple "getting started" base projects.

Java Examples

Getting Started

Getting Started - DataStream API - Skeleton project for a basic Flink Java application using DataStream API
Getting Started - Table API & SQL - Basic Flink Java application using Table API & SQL with DataStream API

Connectors

Kinesis Connectors - Examples of Flink Kinesis Connector source and sink (standard and EFO)
Kinesis Source Deaggregation - Handling Kinesis record deaggregation in the Kinesis source
Kafka Connectors - Examples of Flink Kafka Connector source and sink
Kafka Config Providers - Examples of using Kafka Config Providers for secure configuration management
DynamoDB Stream Source - Reading from DynamoDB Streams as a source
Kinesis Firehose Sink - Writing data to Amazon Kinesis Data Firehose
SQS Sink - Writing data to Amazon SQS
Prometheus Sink - Sending metrics to Prometheus
Flink CDC - Change Data Capture examples using Flink CDC
JdbcSink - Writes to a relational database executing upsert statements

Reading and writing files and transactional data lake formats

Iceberg - Working with Apache Iceberg and Amazon S3 Tables
S3 Sink - Writing JSON data to Amazon S3
S3 Avro Sink - Writing Avro format data to Amazon S3
S3 Avro Source - Reading Avro format data from Amazon S3
S3 Parquet Sink - Writing Parquet format data to Amazon S3
S3 Parquet Source - Reading Parquet format data from Amazon S3

Data Formats & Schema Registry

Avro with Glue Schema Registry - Kinesis - Using Avro format with AWS Glue Schema Registry and Kinesis
Avro with Glue Schema Registry - Kafka - Using Avro format with AWS Glue Schema Registry and Kafka

Stream Processing Patterns

Serialization - Serialization of record and state
Windowing - Time-based window aggregation examples
Side Outputs - Using side outputs for data routing and filtering
Async I/O - Asynchronous I/O patterns with retries for external API calls
Custom Metrics - Creating and publishing custom application metrics
Fetching credentials from Secrets Manager - Dynamically fetching credentials from AWS Secrets Manager

Utilities

Fink Data Generator (JSON) - How to use a Flink application as data generator, for functional and load testing.

Python Examples

Getting Started

Getting Started - Basic PyFlink application Table API & SQL

Handling Python dependencies

Python Dependencies - Managing Python dependencies in PyFlink applications using requirements.txt
Packaged Python Dependencies - Managing Python dependencies packaged with the PyFlink application at build time

Connectors

Datastream Kafka Connector - Using Kafka connector with PyFlink DataStream API
Kafka Config Providers - Secure configuration management for Kafka in PyFlink
S3 Sink - Writing data to Amazon S3 using PyFlink
Firehose Sink - Writing data to Amazon Kinesis Data Firehose
Iceberg Sink - Writing data to Apache Iceberg tables
Hudi Sink - Writing data to Apache Hudi tables

Stream Processing Patterns

Windowing - Time-based window aggregation examples with PyFlink/SQL
User Defined Functions (UDF) - Creating and using custom functions in PyFlink

Utilities

Data Generator - Python script for generating sample data to Kinesis Data Streams
Local Development on Apple Silicon - Setup guide for local development of Flink 1.15 on Apple Silicon Macs (not required with Flink 1.18 or later)

Scala Examples

Getting Started

Getting Started - DataStream API - Skeleton project for a basic Flink Scala application using DataStream API

Infrastructure & Operations

Auto Scaling - Custom autoscaler for Amazon Managed Service for Apache Flink
Scheduled Scaling - Scale applications up and down based on daily time schedules
Monitoring - Extended CloudWatch Dashboard examples for monitoring applications
Scripts - Useful shell scripts for interacting with Amazon Managed Service for Apache Flink control plane API

Contributing

See Contributing Guidelines for more information.

License Summary

This sample code is made available under the MIT-0 license. See the LICENSE file.

Core symbols most depended-on inside this repo

getTicker

called by 10

java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java

property_map

called by 8

python/Windowing/main.py

equals

called by 8

java/JdbcSink/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java

getPrice

called by 7

java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java

hashCode

called by 6

java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java

getEventTime

called by 5

java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java

toString

called by 5

java/Iceberg/IcebergSQLSink/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java

hashCode

called by 5

java/JdbcSink/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java

Shape

Method 501

Class 129

Function 43

Enum 4

Route 3

Languages

Java91%

Python7%

TypeScript2%

Modules by API surface

java/Serialization/CustomTypeInfo/src/test/java/com/amazonaws/services/msf/domain/MoreKryoSerializationExamplesTest.java34 symbols

java/Serialization/CustomTypeInfo/src/main/java/com/amazonaws/services/msf/domain/VehicleEvent.java17 symbols

java/Serialization/CustomTypeInfo/src/main/java/com/amazonaws/services/msf/domain/AggregateVehicleEvent.java15 symbols

java/Windowing/src/main/java/com/amazonaws/services/msf/windowing/WindowStreamingJob.java12 symbols

java/PrometheusSink/src/main/java/com/amazonaws/services/msf/domain/TemperatureSample.java11 symbols

java/KafkaConnectors/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java11 symbols

java/JdbcSink/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java11 symbols

java/Iceberg/IcebergSQLSink/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java11 symbols

java/Iceberg/IcebergSQLSink/src/main/java/com/amazonaws/services/msf/IcebergSQLSinkJob.java11 symbols

java/FlinkDataGenerator/src/main/java/com/amazonaws/services/msf/domain/StockPrice.java11 symbols

java/DynamoDBStreamSource/src/main/java/com/amazonaws/services/msf/ChangeEvent.java11 symbols

java/DynamoDBStreamSource/src/main/java/com/amazonaws/services/msf/DdbTableItem.java10 symbols

Datastores touched

(mysql)Database · 1 repos

targetdbDatabase · 1 repos

testdbDatabase · 1 repos

For agents

$ claude mcp add amazon-managed-service-for-apache-flink-examples \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact

github.com/aws-samples/amazon-managed-service-for-apache-flink-examples @main

Amazon Managed Service for Apache Flink Examples

Table of Contents

Java Examples

Getting Started

Connectors

Reading and writing files and transactional data lake formats

Data Formats & Schema Registry

Stream Processing Patterns

Utilities

Python Examples

Getting Started

Handling Python dependencies

Connectors

Stream Processing Patterns

Utilities

Scala Examples

Getting Started

Infrastructure & Operations

Contributing

License Summary

Core symbols most depended-on inside this repo

Shape

Languages

Modules by API surface

Datastores touched

For agents

Ask about this repo answers extend the page