NVIDIA device plugin for Kubernetes

Table of Contents

• About
• Prerequisites
• Quick Start
• Preparing your GPU Nodes
  • Example for debian-based systems with docker and containerd
  • Install the NVIDIA Container Toolkit
  • Notes on CRI-O configuration
• Enabling GPU Support in Kubernetes
• Running GPU Jobs
• Configuring the NVIDIA device plugin binary
• As command line flags or envvars
• As a configuration file
• Configuration Option Details
• Shared Access to GPUs
  • With CUDA Time-Slicing
  • With CUDA MPS
• IMEX Support
• Catalog of Labels
• Deployment via helm
• Configuring the device plugin's helm chart
  • Passing configuration to the plugin via a ConfigMap
  • Single Config File Example
  • Multiple Config File Example
  • Updating Per-Node Configuration With a Node Label
  • Setting other helm chart values
  • Deploying with gpu-feature-discovery for automatic node labels
  • Deploying gpu-feature-discovery in standalone mode
• Deploying via helm install with a direct URL to the helm package
• Building and Running Locally
• With Docker
  • Build
  • Run
• Without Docker
  • Build
  • Run
• Changelog
• Issues and Contributing
• Versioning
• Upgrading Kubernetes with the Device Plugin

About

The NVIDIA device plugin for Kubernetes is a Daemonset that allows you to automatically:

• Expose the number of GPUs on each nodes of your cluster
• Keep track of the health of your GPUs
• Run GPU enabled containers in your Kubernetes cluster.

This repository contains NVIDIA's official implementation of the Kubernetes device plugin. As of v0.15.0 this repository also holds the implementation for GPU Feature Discovery labels, for further information on GPU Feature Discovery see here.

Please note that:

• The NVIDIA device plugin API is beta as of Kubernetes v1.10.
• The NVIDIA device plugin is currently lacking
• Comprehensive GPU health checking features
• GPU cleanup features
• Support will only be provided for the official NVIDIA device plugin (and not for forks or other variants of this plugin).

Prerequisites

The list of prerequisites for running the NVIDIA device plugin is described below:

• NVIDIA drivers ~= 384.81
• nvidia-docker >= 2.0 || nvidia-container-toolkit >= 1.7.0 (>= 1.11.0 to use integrated GPUs on Tegra-based systems)
• nvidia-container-runtime configured as the default low-level runtime
• Kubernetes version >= 1.10

Quick Start

Preparing your GPU Nodes

The following steps need to be executed on all your GPU nodes. This README assumes that the NVIDIA drivers and the nvidia-container-toolkit have been pre-installed. It also assumes that you have configured the nvidia-container-runtime as the default low-level runtime to use.

Please see: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html

Example for debian-based systems with docker and containerd

Install the NVIDIA Container Toolkit

For instructions on installing and getting started with the NVIDIA Container Toolkit, refer to the installation guide.

Also note the configuration instructions for:

• containerd
• CRI-O
• docker (Deprecated)

Remembering to restart each runtime after applying the configuration changes.

If the nvidia runtime should be set as the default runtime (with non-cri docker versions, for example), the --set-as-default argument must also be included in the commands above. If this is not done, a RuntimeClass needs to be defined:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia

Notes on CRI-O configuration

When running kubernetes with CRI-O, add the config file to set the nvidia-container-runtime as the default low-level OCI runtime under /etc/crio/crio.conf.d/99-nvidia.conf. This will take priority over the default crun config file at /etc/crio/crio.conf.d/10-crun.conf:

[crio]

  [crio.runtime]
    default_runtime = "nvidia"

    [crio.runtime.runtimes]

      [crio.runtime.runtimes.nvidia]
        runtime_path = "/usr/bin/nvidia-container-runtime"
        runtime_type = "oci"

As stated in the linked documentation, this file can automatically be generated with the nvidia-ctk command:

sudo nvidia-ctk runtime configure --runtime=crio --set-as-default --config=/etc/crio/crio.conf.d/99-nvidia.conf

CRI-O uses crun as default low-level OCI runtime so crun needs to be added to the runtimes of the nvidia-container-runtime in the config file at /etc/nvidia-container-runtime/config.toml:

[nvidia-container-runtime]
runtimes = ["crun", "docker-runc", "runc"]

And then restart CRI-O:

sudo systemctl restart crio

Enabling GPU Support in Kubernetes

Once you have configured the options above on all the GPU nodes in your cluster, you can enable GPU support by deploying the following Daemonset:

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml

Note: This is a simple static daemonset meant to demonstrate the basic features of the nvidia-device-plugin. Please see the instructions below for Deployment via helm when deploying the plugin in a production setting.

Running GPU Jobs

With the daemonset deployed, NVIDIA GPUs can now be requested by a container using the nvidia.com/gpu resource type:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  restartPolicy: Never
  containers:
    - name: cuda-container
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
      resources:
        limits:
          nvidia.com/gpu: 1 # requesting 1 GPU
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
EOF

$ kubectl logs gpu-pod
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

[!WARNING] If you do not request GPUs when you use the device plugin, the plugin exposes all the GPUs on the machine inside your container.

Configuring the NVIDIA device plugin binary

The NVIDIA device plugin has a number of options that can be configured for it. These options can be configured as command line flags, environment variables, or via a config file when launching the device plugin. Here we explain what each of these options are and how to configure them directly against the plugin binary. The following section explains how to set these configurations when deploying the plugin via helm.

As command line flags or envvars

As a configuration file

version: v1
flags:
  migStrategy: "none"
  failOnInitError: true
  nvidiaDriverRoot: "/"
  plugin:
    passDeviceSpecs: false
    deviceListStrategy: "envvar"
    deviceIDStrategy: "uuid"

Note: The configuration file has an explicit plugin section because it is a shared configuration between the plugin and gpu-feature-discovery. All options inside the plugin section are specific to the plugin. All options outside of this section are shared.

Configuration Option Details

MIG_STRATEGY: the desired strategy for exposing MIG devices on GPUs that support it

[none | single | mixed] (default 'none')

The MIG_STRATEGY option configures the daemonset to be able to expose Multi-Instance GPUs (MIG) on GPUs that support them. More information on what these strategies are and how they should be used can be found in Supporting Multi-Instance GPUs (MIG) in Kubernetes.

Note: With a MIG_STRATEGY of mixed, you will have additional resources available to you of the form nvidia.com/mig-<slice_count>g.<memory_size>gb that you can set in your pod spec to get access to a specific MIG device.

FAIL_ON_INIT_ERROR: fail the plugin if an error is encountered during initialization, otherwise block indefinitely

(default 'true')

When set to true, the FAIL_ON_INIT_ERROR option fails the plugin if an error is encountered during initialization. When set to false, it prints an error message and blocks the plugin indefinitely instead of failing. Blocking indefinitely follows legacy semantics that allow the plugin to deploy successfully on nodes that don't have GPUs on them (and aren't supposed to have GPUs on them) without throwing an error. In this way, you can blindly deploy a daemonset with the plugin on all nodes in your cluster, whether they have GPUs on them or not, without encountering an error. However, doing so means that there is no way to detect an actual error on nodes that are supposed to have GPUs on them. Failing if an initialization error is encountered is now the default and should be adopted by all new deployments.

NVIDIA_DRIVER_ROOT: the root path for the NVIDIA driver installation

(default '/')

When the NVIDIA drivers are installed directly on the host, this should be set to '/'. When installed elsewhere (e.g. via a driver container), this should be set to the root filesystem where the drivers are installed (e.g. '/run/nvidia/driver').

Note: This option is only necessary when used in conjunction with the $PASS_DEVICE_SPECS option described below. It tells the plugin what prefix to add to any device file paths passed back as part of the device specs.

PASS_DEVICE_SPECS: pass the paths and desired device node permissions for any NVIDIA devices being allocated to the container

(default 'false')

This option exists for the sole purpose of allowing the device plugin to interoperate with the CPUManager in Kubernetes. Setting this flag also requires one to deploy the daemonset with elevated privileges, so only do so if you know you need to interoperate with the CPUManager.

DEVICE_LIST_STRATEGY: the desired strategy for passing the device list to the underlying runtime

[envvar | volume-mounts | cdi-annotations | cdi-cri ] (default 'envvar')

Note: Multiple device list strategies can be specified (as a comma-separated list).

The DEVICE_LIST_STRATEGY flag allows one to choose which strategy the plugin will use to advertise the list of GPUs allocated to a container. Possible values are:

• envvar (default): the NVIDIA_VISIBLE_DEVICES environment variable as described here is used to select the devices that are to be injected by the NVIDIA Container Runtime.
• volume-mounts: the list of devices is passed as a set of volume mounts instead of as an environment variable to instruct the NVIDIA Container Runtime to inject the de