hub / github.com/airbnb/kafka-statsd-metrics2

github.com/airbnb/kafka-statsd-metrics2 @0.5.0

Chat with this repo

repository ↗ · DeepWiki ↗ · release 0.5.0 ↗ · + Follow

133 symbols 388 edges 20 files 3 documented · 2% ● updated 5y ago★ 13417 open issues

What it actually does AI analysis from the code graph — generated when you open this

loading…

README

kafka-statsd-metrics2

Send Kafka Metrics to StatsD.

Contact

Let us know! If you fork this, or if you use it, or if it helps in anyway, we'd love to hear from you! opensource@airbnb.com

What is it about?

Kafka uses Yammer Metrics (now part of the Dropwizard project) for metrics reporting in both the server and the client. This can be configured to report stats using pluggable stats reporters to hook up to your monitoring system.

This project provides a simple integration between Kafka and a StatsD reporter for Metrics.

Metrics can be filtered based on the metric name and the metric dimensions (min, max, percentiles, etc).

Supported Kafka versions

For Kafka 0.9.0.0 or later use kafka-statsd-metrics2-0.5.0
For Kafka 0.8.2.0 or later use kafka-statsd-metrics2-0.4.0
For Kafka 0.8.1.1 or prior use kafka-statsd-metrics2-0.3.0

Releases

0.5.0

0.5.0 add support to report new producer/consumer metrics in kafka-0.9
Compatible with Kafka 0.8
A complete list of all the metrics supported in the metrics reporter can be found here

0.4.0

0.4.0 adds support for tags on metrics. See dogstatsd extensions. If your statsd server does not support tags, you can disable them in the Kafka configuration. See property external.kafka.statsd.tag.enabled below.
The statsd client is com.indeed:java-dogstatsd-client:2.0.11.
support new MetricNames introduced by kafka 0.8.2.x

0.3.0

initial release

How to install?

Download or build the shadow jar for kafka-statsd-metrics.
Install the jar in Kafka classpath, typically ./kafka_2.11-0.9.0.1/libs/
In the Kafka config file, server.properties, add the following properties. Default values are in parenthesis.

How to use metrics in Kafka 0.9 / 0.8?

New metrics in kafka 0.9

Add metric.reporters in producer.properties or consumer.properties

    # declare the reporter if new producer/consumer is used
    metric.reporters=com.airbnb.kafka.kafka09.StatsdMetricsReporter

Run new-producer or new-consumer

Producer:

    bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties
    bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --new-consumer --zookeeper localhost:2181 --topic test -from-beginning --consumer.config config/consumer.properties

Old metrics in kafka 0.8

Add kafka.metrics.reporters in producer.properties or consumer.properties

    # declare the reporter if old producer/consumer is used
    kafka.metrics.reporters=com.airbnb.kafka.kafka08.StatsdMetricsReporter

Run old-producer or old-consumer

    bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties --old-producer
    bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --zookeeper localhost:2181 --topic test -from-beginning --consumer.config config/consumer.properties

Run old-consumer

Configurations

    # declare the reporter if new producer/consumer is used
    metric.reporters=com.airbnb.kafka.StatsdMetricsReporter

    # declare the reporter if old producer/consumer is used
    kafka.metrics.reporters=com.airbnb.kafka.kafka08.StatsdMetricsReporter

    # enable the reporter, (false)
    external.kafka.statsd.reporter.enabled=true

    # the host of the StatsD server (localhost)
    external.kafka.statsd.host=localhost

    # the port of the StatsD server (8125)
    external.kafka.statsd.port=8125

    # enable the support of statsd tag extension, e.g. datadog statsd (true)
    external.kafka.statsd.tag.enabled=true

    # a prefix for all metrics names (empty)
    external.kafka.statsd.metrics.prefix=

    # note that the StatsD reporter follows the global polling interval (10)
    # kafka.metrics.polling.interval.secs=10



    # A regex to exclude some metrics
    # Default is: (kafka\.consumer\.FetchRequestAndResponseMetrics.*)|(.*ReplicaFetcherThread.*)|(kafka\.server\.FetcherLagMetrics\..*)|(kafka\.log\.Log\..*)|(kafka\.cluster\.Partition\..*)
    #
    # The metric name is formatted with this template: group.type.scope.name
    #
    # external.kafka.statsd.metrics.exclude_regex=

    #
    # Each metric provides multiple dimensions: min, max, meanRate, etc
    # This might be too much data.
    # It is possible to disable some metric dimensions with the following properties:
    # By default all dimenstions are enabled.
    #
    # external.kafka.statsd.dimension.enabled.count=true
    # external.kafka.statsd.dimension.enabled.meanRate=true
    # external.kafka.statsd.dimension.enabled.rate1m=true
    # external.kafka.statsd.dimension.enabled.rate5m=true
    # external.kafka.statsd.dimension.enabled.rate15m=true
    # external.kafka.statsd.dimension.enabled.min=true
    # external.kafka.statsd.dimension.enabled.max=true
    # external.kafka.statsd.dimension.enabled.mean=true
    # external.kafka.statsd.dimension.enabled.stddev=true
    # external.kafka.statsd.dimension.enabled.median=true
    # external.kafka.statsd.dimension.enabled.p75=true
    # external.kafka.statsd.dimension.enabled.p95=true
    # external.kafka.statsd.dimension.enabled.p98=true
    # external.kafka.statsd.dimension.enabled.p99=true
    # external.kafka.statsd.dimension.enabled.p999=true

finally restart the Kafka server

How to test your configuration?

You can check your configuration in different ways:

During Kafka startup, the reporter class will be instantiated and initialized. The logs should contain a message similar to: "Kafka Statsd metrics reporter is enabled"
A JMX MBean named kafka:type=com.airbnb.kafka.kafka08.StatsdMetricsReporter should also exist.
Check the logs of your StatsD server
Finally, on the configured StatsD host, you could listen on the configured port and check for incoming data:

    # assuming the Statsd server has been stopped...
    $ nc -ul 8125

    kafka.controller.ControllerStats.LeaderElectionRateAndTimeMs.samples:1|gkafka.controller.ControllerStats
    .LeaderElectionRateAndTimeMs.meanRate:0.05|gkafka.controller.ControllerStats.LeaderElectionRateAndTimeMs.
    1MinuteRate:0.17|gkafka.controller.ControllerStats.LeaderElectionRateAndTimeMs.5MinuteRate:0.19|g....

Sample file of metrics output from statsd

new-producer-metrics.txt

new-consumer-metrics.txt

List of metrics for Kafka 0.8.2

Below are the metrics in Kafka 0.8.2

```bash +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Metrics kind | Metric Name | Metric Tags | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.server.ReplicaManager.LeaderCount | | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.server.ReplicaManager.PartitionCount | | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.server.ReplicaManager.UnderReplicatedPartitions | | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.controller.KafkaController.ActiveControllerCount | | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.controller.KafkaController.OfflinePartitionsCount | | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.controller.KafkaController.PreferredReplicaImbalanceCount | | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.network.RequestChannel.RequestQueueSize | | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.server.ReplicaFetcherManager.Replica_MaxLag | {"clientId" -> clientId} | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.server.ReplicaFetcherManager.Replica_MinFetchRate | {"clientId" -> clientId} | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.server.FetchRequestPurgatory.PurgatorySize | | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.server.FetchRequestPurgatory.NumDelayedRequests | | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.server.ProducerRequestPurgatory.PurgatorySize | | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.server.ProducerRequestPurgatory.NumDelayedRequests | | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.consumer.ConsumerFetcherManager.MaxLag | {"clientId" -> clientId} | +--------------+-----------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ | Gauge | kafka.consumer.ConsumerFetcherManager.MinFetchRate | {"clientId" -> clientId} | +--------------+-----------------------------------------------------------------------

Extension points exported contracts — how you extend this code

StatsdMetricsReporterMBean (Interface)

@see kafka.metrics.KafkaMetricsReporterMBean: the custom reporter needs to additionally implement an MBean trait that ex [1 …

src/main/java/com/airbnb/kafka/kafka08/StatsdMetricsReporterMBean.java

Core symbols most depended-on inside this repo

getName

called by 16

src/main/java/com/airbnb/metrics/Parser.java

getTags

called by 11

src/main/java/com/airbnb/metrics/Parser.java

isRunning

called by 8

src/main/java/com/airbnb/kafka/kafka09/StatsdMetricsReporter.java

matches

called by 7

src/main/java/com/airbnb/metrics/ExcludeMetricPredicate.java

send

called by 6

src/main/java/com/airbnb/metrics/StatsDReporter.java

parse

called by 6

src/main/java/com/airbnb/metrics/ParserForTagInMBeanName.java

format

called by 5

src/main/java/com/airbnb/metrics/MetricNameFormatter.java

getMBeanName

called by 5

src/main/java/com/airbnb/kafka/kafka08/StatsdMetricsReporter.java

Shape

Method 111

Class 20

Enum 1

Interface 1

Languages

Java100%

Modules by API surface

src/test/java/com/airbnb/metrics/StatsDReporterTest.java32 symbols

src/main/java/com/airbnb/metrics/StatsDReporter.java15 symbols

src/main/java/com/airbnb/kafka/kafka09/StatsdMetricsReporter.java11 symbols

src/test/java/com/airbnb/metrics/ParserTest.java8 symbols

src/main/java/com/airbnb/kafka/kafka08/StatsdMetricsReporter.java8 symbols

src/main/java/com/airbnb/metrics/StatsDMetricsRegistry.java7 symbols

src/main/java/com/airbnb/metrics/KafkaStatsDReporter.java7 symbols

src/test/java/com/airbnb/metrics/KafkaStatsDReporterTest.java6 symbols

src/test/java/com/airbnb/kafka/kafka08/StatsdMetricsReporterTest.java5 symbols

src/main/java/com/airbnb/metrics/Dimension.java5 symbols

src/test/java/com/airbnb/kafka/kafka09/StatsdMetricsReporterTest.java4 symbols

src/main/java/com/airbnb/metrics/Parser.java4 symbols

For agents

$ claude mcp add kafka-statsd-metrics2 \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact