node-problem-detector

node-problem-detector aims to make various node problems visible to the upstream layers in the cluster management stack. It is a daemon that runs on each node, detects node problems and reports them to apiserver. node-problem-detector can either run as a DaemonSet or run standalone. Now it is running as a Kubernetes Addon enabled by default in the GKE cluster. It is also enabled by default in AKS as part of the AKS Linux Extension.

Background

There are tons of node problems that could possibly affect the pods running on the node, such as: * Infrastructure daemon issues: ntp service down; * Hardware issues: Bad CPU, memory or disk; * Kernel issues: Kernel deadlock, corrupted file system; * Container runtime issues: Unresponsive runtime daemon; * ...

Currently, these problems are invisible to the upstream layers in the cluster management stack, so Kubernetes will continue scheduling pods to the bad nodes.

To solve this problem, we introduced this new daemon node-problem-detector to collect node problems from various daemons and make them visible to the upstream layers. Once upstream layers have visibility to those problems, we can discuss the remedy system.

Problem API

node-problem-detector uses Event and NodeCondition to report problems to apiserver. * NodeCondition: Permanent problem that makes the node unavailable for pods should be reported as NodeCondition. * Event: Temporary problem that has limited impact on pod but is informative should be reported as Event.

Problem Daemon

A problem daemon is a sub-daemon of node-problem-detector. It monitors specific kinds of node problems and reports them to node-problem-detector.

A problem daemon could be: * A tiny daemon designed for dedicated Kubernetes use-cases. * An existing node health monitoring daemon integrated with node-problem-detector.

Currently, a problem daemon is running as a goroutine in the node-problem-detector binary. In the future, we'll separate node-problem-detector and problem daemons into different containers, and compose them with pod specification.

Each category of problem daemon can be disabled at compilation time by setting corresponding build tags. If they are disabled at compilation time, then all their build dependencies, global variables and background goroutines will be trimmed out of the compiled executable.

List of supported problem daemons types:

Exporter

An exporter is a component of node-problem-detector. It reports node problems and/or metrics to certain backends. Some of them can be disabled at compile-time using a build tag. List of supported exporters:

Usage

Flags

• --version: Print current version of node-problem-detector.
• --hostname-override: A customized node name used for node-problem-detector to update conditions and emit events. node-problem-detector gets node name first from hostname-override, then NODE_NAME environment variable and finally fall back to os.Hostname.

For System Log Monitor

• --config.system-log-monitor: List of paths to system log monitor configuration files, comma-separated, e.g. config/kernel-monitor.json. Node problem detector will start a separate log monitor for each configuration. You can use different log monitors to monitor different system logs.

For System Stats Monitor

• --config.system-stats-monitor: List of paths to system stats monitor config files, comma-separated, e.g. config/system-stats-monitor.json. Node problem detector will start a separate system stats monitor for each configuration. You can use different system stats monitors to monitor different problem-related system stats.

For Custom Plugin Monitor

• --config.custom-plugin-monitor: List of paths to custom plugin monitor config files, comma-separated, e.g. config/custom-plugin-monitor.json. Node problem detector will start a separate custom plugin monitor for each configuration. You can use different custom plugin monitors to monitor different node problems.

For Health Checkers

Health checkers are configured as custom plugins, using the config/health-checker-*.json config files.

For Kubernetes exporter

• --enable-k8s-exporter: Enables reporting to Kubernetes API server, default to true.
• --apiserver-override: A URI parameter used to customize how node-problem-detector connects the apiserver. This is ignored if --enable-k8s-exporter is false. The format is the same as the source flag of Heapster. For example, to run without auth, use the following config: http://APISERVER_IP:APISERVER_PORT?inClusterConfig=false Refer to heapster docs for a complete list of available options.
• --address: The address to bind the node problem detector server.
• --port: The port to bind the node problem detector server. Use 0 to disable.

For Prometheus exporter

• --prometheus-address: The address to bind the Prometheus scrape endpoint, default to 127.0.0.1.
• --prometheus-port: The port to bind the Prometheus scrape endpoint, default to 20257. Use 0 to disable.

For Stackdriver exporter

• --exporter.stackdriver: Path to a Stackdriver exporter config file, e.g. config/exporter/stackdriver-exporter.json, defaults to empty string. Set to empty string to disable.

Deprecated Flags

• --system-log-monitors: List of paths to system log monitor config files, comma-separated. This option is deprecated, replaced by --config.system-log-monitor, and will be removed. NPD will panic if both --system-log-monitors and --config.system-log-monitor are set.

• --custom-plugin-monitors: List of paths to custom plugin monitor config files, comma-separated. This option is deprecated, replaced by --config.custom-plugin-monitor, and will be removed. NPD will panic if both --custom-plugin-monitors and --config.custom-plugin-monitor are set.

Build Image

• Install development dependencies for libsystemd and the ARM GCC toolchain

• Debian/Ubuntu: apt install libsystemd-dev gcc-aarch64-linux-gnu

• git clone git@github.com:kubernetes/node-problem-detector.git

• Run make in the top directory. It will:

• Build the binary.
• Build the docker image. The binary and config/ are copied into the docker image.

If you do not need certain categories of problem daemons, you could choose to disable them at compilation time. This is the best way of keeping your node-problem-detector runtime compact without unnecessary code (e.g. global variables, goroutines, etc). You can do so via setting the BUILD_TAGS environment variable before running make. For example:

BUILD_TAGS="disable_custom_plugin_monitor disable_system_stats_monitor" make

The above command will compile the node-problem-detector without Custom Plugin Monitor and System Stats Monitor. Check out the Problem Daemon section to see how to disable each problem daemon during compilation time.

Push Image

make push uploads the docker image to a registry. By default, the image will be uploaded to staging-k8s.gcr.io. It's easy to modify the Makefile to push the image to another registry.

Installation

The easiest way to install node-problem-detector into your cluster is to use the Helm chart:

helm repo add deliveryhero https://charts.deliveryhero.io/
helm install --generate-name deliveryhero/node-problem-detector

Alternatively, to install node-problem-detector manually:

1. Edit node-problem-detector.yaml to fit your environment. Set log volume to your system log directory (used by SystemLogMonitor). You can use a ConfigMap to overwrite the config directory inside the pod.

2. Edit node-problem-detector-config.yaml to configure node-problem-detector.

3. Edit rbac.yaml to fit your environment.

4. Create the ServiceAccount and ClusterRoleBinding with kubectl create -f rbac.yaml.

5. Create the ConfigMap with kubectl create -f node-problem-detector-config.yaml.

6. Create the DaemonSet with kubectl create -f node-problem-detector.yaml.

Start Standalone

To run node-problem-detector standalone, you should set inClusterConfig to false and teach node-problem-detector how to access apiserver with apiserver-override.

To run node-problem-detector standalone with an insecure apiserver connection:

node-problem-detector --apiserver-override=http://APISERVER_IP:APISERVER_INSECURE_PORT?inClusterConfig=false

For more scenarios, see here

Windows

Node Problem Detector has preliminary support Windows. Most of the functionality has not been tested but filelog plugin works.

Follow Issue #461 for development status of Windows support.

Development

To develop NPD on Window