Overview

kube-state-metrics (KSM) is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. (See examples in the Metrics section below.) It is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods.

kube-state-metrics is about generating metrics from Kubernetes API objects without modification. This ensures that features provided by kube-state-metrics have the same grade of stability as the Kubernetes API objects themselves. In turn, this means that kube-state-metrics in certain situations may not show the exact same values as kubectl, as kubectl applies certain heuristics to display comprehensible messages. kube-state-metrics exposes raw data unmodified from the Kubernetes API, this way users have all the data they require and perform heuristics as they see fit.

The metrics are exported on the HTTP endpoint /metrics on the listening port (default 8080). They are served as plaintext. They are designed to be consumed either by Prometheus itself or by a scraper that is compatible with scraping a Prometheus client endpoint. You can also open /metrics in a browser to see the raw metrics. Note that the metrics exposed on the /metrics endpoint reflect the current state of the Kubernetes cluster. When Kubernetes objects are deleted they are no longer visible on the /metrics endpoint.

[!NOTE] This README is generated from a template. Please make your changes there and run make generate-template.

Table of Contents

• Versioning
• Kubernetes Version
• Compatibility matrix
• Resource group version compatibility
• Container Image
• Metrics Documentation
• ECMAScript regular expression support for allow and deny lists
• Conflict resolution in label names
• Kube-state-metrics self metrics
• kube-state-metrics vs. metrics-server
• Scaling kube-state-metrics
• Resource recommendation
• Latency
• A note on costing
• Horizontal sharding
  • Automated sharding
  • Deployment sharding
• Daemonset sharding for pod metrics
• Resource filtering
• Setup
• Building the Docker container
• Usage
• Kubernetes Deployment
• Limited privileges environment
• Helm Chart
• Development
• Developer Contributions
• Community

Versioning

Kubernetes Version

kube-state-metrics uses client-go to talk with Kubernetes clusters. The supported Kubernetes cluster version is determined by client-go. All additional compatibility is only best effort, or happens to still/already be supported.

Compatibility matrix

At most, 5 kube-state-metrics and 5 kubernetes releases will be recorded below. Generally, it is recommended to use the latest release of kube-state-metrics. If you run a very recent version of Kubernetes, you might want to use an unreleased version to have the full range of supported resources. If you run an older version of Kubernetes, you might need to run an older version in order to have full support for all resources. Be aware, that the maintainers will only support the latest release. Older versions might be supported by interested users of the community.

Resource group version compatibility

Resources in Kubernetes can evolve, i.e., the group version for a resource may change from alpha to beta and finally GA in different Kubernetes versions. For now, kube-state-metrics will only use the oldest API available in the latest release.

Container Image

The latest container image can be found at:

• registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.0 (arch: amd64, arm, arm64, ppc64le and s390x)
• Multi-architecture images

Metrics Documentation

Any resources and metrics based on alpha Kubernetes APIs are excluded from any stability guarantee, which may be changed at any given release.

See the docs directory for more information on the exposed metrics.

Conflict resolution in label names

The *_labels family of metrics exposes Kubernetes labels as Prometheus labels. As Kubernetes is more liberal than Prometheus in terms of allowed characters in label names, we automatically convert unsupported characters to underscores. For example, app.kubernetes.io/name becomes label_app_kubernetes_io_name.

This conversion can create conflicts when multiple Kubernetes labels like foo-bar and foo_bar would be converted to the same Prometheus label label_foo_bar.

Kube-state-metrics automatically adds a suffix _conflictN to resolve this conflict, so it converts the above labels to label_foo_bar_conflict1 and label_foo_bar_conflict2.

If you'd like to have more control over how this conflict is resolved, you might want to consider addressing this issue on a different level of the stack, e.g. by standardizing Kubernetes labels using an Admission Webhook that ensures that there are no possible conflicts.

ECMAScript regular expression support for allow and deny lists

Starting from #2616, kube-state-metrics supports ECMAScript's regexp for allow and deny lists. This was incorporated as a workaround for the limitations of the regexp package in Go, which does not support lookarounds due to their non-linear time complexity. Please note that while lookarounds are now supported for allow and deny lists, regular expressions' evaluation time is capped at a minute to prevent performance issues.

Kube-state-metrics self metrics

kube-state-metrics exposes its own general process metrics under --telemetry-host and --telemetry-port (default 8081).

kube-state-metrics also exposes list and watch success and error metrics. These can be used to calculate the error rate of list or watch resources. If you encounter those errors in the metrics, it is most likely a configuration or permission issue, and the next thing to investigate would be looking at the logs of kube-state-metrics.

Example of the above mentioned metrics:

kube_state_metrics_list_total{resource="*v1.Node",result="success"} 1
kube_state_metrics_list_total{resource="*v1.Node",result="error"} 52
kube_state_metrics_watch_total{resource="*v1beta1.Ingress",result="success"} 1

kube-state-metrics also exposes some http request metrics, examples of those are:

http_request_duration_seconds_bucket{handler="metrics",method="get",le="2.5"} 30
http_request_duration_seconds_bucket{handler="metrics",method="get",le="5"} 30
http_request_duration_seconds_bucket{handler="metrics",method="get",le="10"} 30
http_request_duration_seconds_bucket{handler="metrics",method="get",le="+Inf"} 30
http_request_duration_seconds_sum{handler="metrics",method="get"} 0.021113919999999998
http_request_duration_seconds_count{handler="metrics",method="get"} 30

kube-state-metrics also exposes build and configuration metrics:

kube_state_metrics_build_info{branch="main",goversion="go1.15.3",revision="6c9d775d",version="v2.0.0-beta"} 1
kube_state_metrics_shard_ordinal{shard_ordinal="0"} 0
kube_state_metrics_total_shards 1

kube_state_metrics_build_info is used to expose version and other build information. For more usage about the info pattern, please check this blog post. Sharding metrics expose --shard and --total-shards flags and can be used to validate run-time configuration, see /examples/prometheus-alerting-rules.

kube-state-metrics also exposes metrics about it config file and the Custom Resource State config file:

kube_state_metrics_config_hash{filename="crs.yml",type="customresourceconfig"} 2.38272279311849e+14
kube_state_metrics_config_hash{filename="config.yml",type="config"} 2.65285922340846e+14
kube_state_metrics_last_config_reload_success_timestamp_seconds{filename="crs.yml",type="customresourceconfig"} 1.6704882592037103e+09
kube_state_metrics_last_config_reload_success_timestamp_seconds{filename="config.yml",type="config"} 1.6704882592035313e+09
kube_state_metrics_last_config_reload_successful{filename="crs.yml",type="customresourceconfig"} 1
kube_state_metrics_last_config_reload_successful{filename="config.yml",type="config"} 1

kube-state-metrics vs. metrics-server

The metrics-server is a project that has been inspired by Heapster and is implemented to serve the goals of core metrics pipelines in Kubernetes monitoring architecture. It is a cluster level component which periodically scrapes metrics from all Kubernetes nodes served by Kubelet through Metrics API. The metrics are aggregated, stored in memory and served in Metrics API format. The metrics-server stores the latest values only and is not responsible for forwarding metrics to third-party destinations.

kube-state-metrics is focused on generating completely new metrics from Kubernetes' object state (e.g. metrics based on deployments, replica sets, etc.). It holds an entire snapshot of Kubernetes state in memory and continuously generates new metrics based off of it. And just like the metrics-server it too is not responsible for exporting its metrics anywhere.

Having kube-state-metrics as a separate project also enables access to these metrics from monitoring systems such as Prometheus.

Scaling kube-state-metrics

Resource recommendation

Resource usage for kube-state-metrics changes with the Kubernetes objects (Pods/Nodes/Deployments/Secrets etc.) size of the cluster. To some extent, the Kubernetes objects in a cluster are in direct proportion to the node number of the cluster.

As a general rule, you should allocate:

• 250MiB memory
• 0.1 cores

Note that if CPU limits are set too low, kube-state-metrics' internal queues will not be able to be worked off quickly enough, resulting in increased memory consumption as the queue length grows. If you experience problems resulting from high memory allocation or CPU throttling, try increasing the CPU limits.

Latency

In a 100 node cluster scaling test the latency numbers were as follows:

"Perc50": 259615384 ns,
"Perc90": 475000000 ns,
"Perc99": 906666666 ns.

A note on costing

By default, kube-state-metrics exposes several metrics for events across your cluster. If you have a large number of frequently-updating resources on your cluster, you may find that a lot of data is ingested into these metrics. This can incur high costs on some cloud providers. Please take a moment to configure what metrics you'd like to expose, as well as consult the documentation for your Kubernetes environment in order to avoid unexpectedly high costs.

Horizontal sharding

In order to shard kube-state-metrics horizontally, some automated sharding capabilities have been implemented. It is configured with the following flags:

• --shard (zero