Hi! I'm Alisa, a DevOps engineer. This is my translation to English from my original article.

In this article, I'll explain how I set up automatic NVIDIA driver updates for Jupyter and our ML stack in a managed Kubernetes cluster.

Here was the problem: containers and ML libraries were updated more often than the system images on our GPU nodes. As a result, the NVIDIA driver version was incompatible with the CUDA version inside the container. When I ran nvidia-smi, I got the Driver/library version mismatch error, and CUDA could not detect the driver on the host.

I needed to update Jupyter and its ML stack, which depends on CUDA. This also meant updating the NVIDIA drivers on our GPU nodes. I could do it manually on every node, but that was not a good option for me. So I decided to automate the process. That's what this article is about. I'll cover manual driver updates, different ways to automate them, and how we handled conflicts between GPU Operator and preinstalled drivers.

Table of Contents

Our Initial Setup
Updating the NVIDIA Driver Manually
Automating the Process with a DaemonSet
GPU Operator
Conclusion

Our Initial Setup

First, let's look at our configuration.

Cluster: Yandex Managed Service for Kubernetes, managed with Terraform.
GPU nodes: a dedicated node group with NVIDIA Tesla V100 GPUs. To choose a specific GPU model, you need to set the platform type. You can find all available options in the official documentation. In this article, I'll use Intel Cascade Lake with NVIDIA Tesla V100 (gpu-standard-v2) as an example.

To make sure that only specific workloads that need GPUs can run on these nodes, I added a taint: node_taints = ["my-mode-jupyter=true:NoSchedule"].

Here is an example of the full Terraform configuration for a node group:

resource "yandex_kubernetes_node_group" "my-mode-test-v100" {
  cluster_id  = yandex_kubernetes_cluster.<cluster-name>.id
  name        = "my-mode-test-v100"
  version     = "1.28"

  node_labels = { "node.kubernetes.io/role" = "my-mode-jupyter", "inst_type" = "my-mode-large-v100"}
  node_taints = ["my-mode-jupyter=true:NoSchedule"]

  instance_template {
    name = "my-mode-test-v100-8-48-gpu-{instance.short_id}"
    platform_id = "gpu-standard-v2"

    resources {
      memory        = 48
      cores         = 8
      core_fraction = 100
      gpus          = 1
    }

    boot_disk {
      type = "network-ssd"
      size = 96
    }

    scheduling_policy {
      preemptible = true
    }
    metadata = {
      "serial-port-enable" = "1"
      ssh-keys = "<admin>:${<ssh-key>},ubuntu:$${<ssh-key>}"
    }

    gpu_settings {
      gpu_environment = "runc_drivers_cuda"
    }

    network_interface {
      subnet_ids = [yandex_vpc_subnet.<subnet-name>.id]
    }
  }
  scale_policy {
    auto_scale {
      min     = 1
      max     = 3
      initial = 1
    }
  }

  allocation_policy {
    location {
      zone = yandex_vpc_subnet.<subnet-name>.zone
    }
  }

  maintenance_policy {
    auto_upgrade = false
    auto_repair  = true
  }

}

I use JupyterHub on our GPU nodes. When a user starts a session, JupyterHub creates a pod with Jupyter Notebook. Users can choose the pod's resources: with or without a GPU, and with different CPU and RAM combinations.
Each pod uses a container image based on the official jupyter/scipy-notebook image.
I added the ML libraries was need (PyTorch, CatBoost, etc.) for CUDA 12.
When creating GPU nodes in Yandex Cloud, we chose the option with preinstalled NVIDIA drivers (the Node Group setting gpu_environment = "runc_drivers_cuda").

If you need to update just one node without recreating the whole node group and reinstalling all the nodes, you can do it manually. This requires downtime for that node (cordon/drain and reboot), but it does not affect the other nodes or the whole group.

Updating the NVIDIA Driver Manually

When gpu_environment = "runc_drivers_cuda", Yandex Cloud installs the NVIDIA driver inside the VM using NVIDIA's .run installer. You cannot choose the driver version. Driver updates only come with new node images. If you want to use CUDA 12.6, you have to update the driver manually or wait for Yandex Cloud to release an updated image.

Taking the Node Out of Service

kubectl drain <node-name>

After the drain, new pods are no longer scheduled on the node, and Kubernetes moves the existing workloads to other nodes.

A Privileged Pod with Access to the Host Root Filesystem

The following manifest creates a pod in the default namespace (or any other namespace) and mounts the host's root filesystem (/) with read/write access.

apiVersion: v1
kind: Pod
metadata:
  name: privileged-jupyter-pod
spec:
  nodeName: <node-name>          # Run the pod on this specific node
  containers:
  - name: debug
    image: ubuntu:20.04
    command: ["sleep", "infinity"] # Keep the container running until it is stopped
    securityContext:
      privileged: true           # Give the container full access to kernel features. This is dangerous, but needed to work with the host
    volumeMounts:
    - name: host-root
      mountPath: /host           # Mount the host filesystem at /host inside the container
      mountPropagation: HostToContainer # Host mounts are visible in the container, but container mounts are not visible on the host
  volumes:
  - name: host-root
    hostPath:
      path: /                    # Mount the root of the host filesystem
      type: Directory
  hostPID: true                  # Let the container see host PIDs
  hostNetwork: true              # Use the host network

Now we connect to the pod on the node:

kubectl exec -it privileged-jupyter-pod -- chroot /host bash

Checking the Current Installation

I enter my own privileged pod to inspect the node's system and install a new driver:

$ dpkg -l | grep nvidia
libnvidia-container-tools     1.16.2-1    amd64    NVIDIA container runtime library (command-line tools)
libnvidia-container1:amd64    1.16.2-1    amd64    NVIDIA container runtime library
nvidia-container-toolkit      1.16.2-1    amd64    NVIDIA Container toolkit
nvidia-container-toolkit-base 1.16.2-1    amd64    NVIDIA Container Toolkit Base
nvidia-docker2                2.14.0-1    all      NVIDIA Container Toolkit meta-package

There are no nvidia-driver-*, libnvidia-compute-*, nvidia-utils-*, or other driver packages installed through APT. This suggests that the NVIDIA driver was installed in another way, most likely with the official .run installer from the NVIDIA website.

We can also confirm this from the nvidia-smi output (driver version 515.48.07) and the symbolic link /usr/bin/nvidia-uninstall → nvidia-installer.

$ ls -l /usr/bin/nvidia-uninstall
lrwxrwxrwx 1 root root 16 Nov 30  2023 /usr/bin/nvidia-uninstall -> nvidia-installer

In this case, removing the driver with APT will not remove the installed kernel modules and libraries. It may even break the node.

Removing the Old Driver

chroot /host /usr/bin/nvidia-uninstall
# If asked whether to restore the X configuration, choose "No".
# Kubernetes does not use GNOME, KDE, or Xorg here, so there is no reason to restore a configuration that does not exist.

The .run uninstaller unloads the modules and removes files from /usr/lib/x86_64-linux-gnu/. After it finishes, the nvidia module is no longer listed by lsmod.

Installing the New Driver

chroot /host
apt-get update
apt-get install -y curl gnupg2 ca-certificates build-essential dkms wget
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring\_1.1-1\_all.deb
dpkg -i /cuda-keyring_1.1-1_all.deb
apt-get update
apt-get install -y nvidia-driver-525

The cuda-keyring package adds the signing key for the official repository. After that, nvidia-driver-525 installs the required kernel modules, including nvidia-drm, nvidia-uvm, and others.

Rebooting the Node

While the old kernel module is still loaded, nvidia-smi returns an NVML error:

Failed to initialize NVML: Driver/library version mismatch

I run chroot /host systemctl reboot. The pod stops, the node reboots, and kubelet starts again. After the reboot, I check the driver one more time:

nvidia-smi
# Driver Version: 525.147.05   CUDA Version: 12.0

I can return the node to the pool using kubectl uncordon.

Automating the Process with a DaemonSet

The manual update process is quite simple and follows the same steps every time. So, my first idea was to put it into a DaemonSet with a script. This would reduce the amount of manual work when updating drivers on several nodes.

I select the target nodes using labels, taints, and nodeSelector. Each node still has its own downtime, but the process becomes repeatable. This approach also does not require us to recreate the node group.

Full DaemonSet config

apiVersion: apps/v1
kind: DaemonSet                          # Run a pod on each selected node
metadata:
  name: nvidia-driver-upgrade
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-driver-upgrade       # Match the DaemonSet with its pods
  template:
    metadata:
      labels:
        name: nvidia-driver-upgrade     # Label used by the selector above
    spec:
      nodeSelector:
        nvidia-driver-upgrade: "true"   # Run only on nodes with this label (add it before deploying)
      tolerations:
        - key: "my-mode-jupyter"        # Allow the pod on nodes with this taint key
          operator: "Exists"
          effect: "NoSchedule"
      containers:
      - name: upgrade
        image: ubuntu:20.04
        securityContext:
          privileged: true              # Access host features for chroot
        command: [bash, -c, |           # Run the Bash script
          set -eux                      # Trace commands and exit on errors
          if [ -f /host/usr/bin/nvidia-uninstall ]; then
            chroot /host /usr/bin/nvidia-uninstall --silent || true
          fi
          chroot /host apt-get update
          chroot /host apt-get install -y curl gnupg2 ca-certificates build-essential dkms wget
          chroot /host wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring\_1.1-1\_all.deb
          chroot /host apt-get update
          chroot /host apt-get install -y nvidia-driver-525
          chroot /host systemctl reboot
        ]
        volumeMounts:
          - name: host-root
            mountPath: /host
            mountPropagation: HostToContainer   # Host mounts are visible in the container
      volumes:
        - name: host-root
          hostPath:
            path: /                  # Mount the host root in the container
            type: Directory
      hostPID: true                  # See host PIDs
      hostNetwork: true              # Use the host network
      restartPolicy: OnFailure       # Restart only if the process fails

Next, I deploy the DaemonSet to the cluster, and it does the work for too. It looks like I have automated everything, but there are a few problems.

First, every change to the kernel version or repository URL requires us to edit the script. Second, the script does not check whether the CUDA version is compatible with the driver, or whether the library version is compatible with the container runtime. Third, there is no rollback. If the script fails, the node may stay broken until a DevOps engineer fixes it manually.

That's why I started looking for other solutions and found NVIDIA GPU Operator.

GPU Operator

NVIDIA GPU Operator automates driver installation, the device plugin, and monitoring in Kubernetes. However, this approach requires me to recreate the node group when switching to it.

The official Yandex Cloud documentation recommends creating GPU nodes without preinstalled drivers first (gpu_environment = "runc") and then installing the operator. The operator manages the driver lifecycle and installs compatible versions.

In my setup, a user starts a separate GPU node for their Jupyter session. When the session ends the node should be delete to save money. A new node may start with an unsuitable preinstalled driver version, or with no driver at all. I need to bring it to the required state quickly and without manual work. So I chose a GPU Operator setup that fits our infrastructure.

Installing with Helm

helm repo add nvidia https://nvidia.github.io/gpu-operator
helm repo update
helm upgrade --install gpu-operator nvidia/gpu-operator \
  -n gpu-operator --create-namespace \
  -f my-values.yaml

It's important to remember that GPU Operator can manage driver installation only on nodes without preinstalled drivers (with gpu_environment = "runc"). Otherwise, it cannot reinstall the driver.

My custom values.yaml looks like this:

driver:
  enabled: true
  nvidiaDriverCRD:
    enabled: false  # Do not create a driver CR
    deployDefaultCR: true  # Install the default CR
    driverType: gpu
  nodeSelector:
    inst_type: <node-label>  # Apply only to nodes with this label

  kernelModuleType: "auto"  # Automatically select the driver kernel module
  usePrecompiled: false  # Build the driver on the node
  repository: <driver-repo>
  image: <driver-image>
  version: "<driver-version>"
  imagePullPolicy: IfNotPresent
  upgradePolicy:
    autoUpgrade: true  # Enable automatic driver updates
    drain:
      enable: false  # Do not drain the node during updates
    gpuPodDeletion:
      force: true  # Delete old GPU pods
    maxUnavailable: 100%

daemonsets:
  labels: {}
  annotations: {}
  priorityClassName: system-node-critical  # Operator pod priority
  nodeSelector:
    inst_type: <node-label>

  tolerations:
    - key: "<taint-key1>"
      operator: Equal
      value: "<taint-value1>"
      effect: NoSchedule

  updateStrategy: RollingUpdate
  rollingUpdate:
    maxUnavailable: "1"

node-feature-discovery:
  worker:
    nodeSelector:
      inst_type: <node-label>  # Run only on the required nodes

    tolerations:
      - key: nvidia.com/gpu  # Allow scheduling on GPU nodes
        operator: Exists
        effect: NoSchedule

Let's go through each section.

The driver Section

This section controls NVIDIA driver installation on GPU nodes. Without the driver, CUDA and the Kubernetes device plugin will not work. Here we define the driver type, version, and how it is installed and updated.

nvidiaDriverCRD:
- enabled: When set to true, you can create several NVIDIADriver custom resources with different settings for different nodes (for example, different versions or GPU/vGPU types).
- deployDefaultCR: When set to true, a default custom resource without a nodeSelector is created and applies to all GPU nodes. When set to false, you need to create custom resources with selectors yourself. At the same time, driver.nodeSelector controls which nodes run the driver DaemonSet.
kernelModuleType: With auto, the operator chooses the best kernel module type based on the GPU and driver version.
usePrecompiled (true/false):
- false: The driver is compiled directly on the node inside a container.
- true: A prebuilt driver image is used, which makes installation faster.
autoUpgrade (true/false):
- true: Enables the driver upgrade controller.
- false: The upgrade settings are ignored, updates are not performed, and the nvidia.com/gpu-driver-upgrade-state label does not change. Special system pods set these labels to show the node's current state. This option is useful when you need to pause upgrades for a while.
gpuPodDeletion.force: When set to true, all GPU pods are forcefully deleted. This allows the old driver kernel modules to be removed before installing a new version.

The daemonsets Section

This section defines the settings for the DaemonSets that run driver containers on GPU nodes.

The priorityClassName: system-node-critical setting makes sure that driver pods can be scheduled even when the cluster is under heavy load.

The node-feature-discovery Section

This component automatically detects a node's hardware and system information (GPU, PCI devices, operating system, kernel version, etc.). It then adds the right labels so that driver DaemonSets can target the correct nodes.

Conclusion

If your infrastructure is still small enough that driver updates do not need their own automation, manual updates can work well. But if you have many GPU nodes for ML projects, I would recommend using GPU Operator, as Yandex Cloud support also recommends. There is just one important thing to keep in mind.

GPU Operator becomes the single source of truth for the driver on each Yandex Cloud node. If a driver is already installed on the host, installing the operator directly can cause version conflicts and leave old kernel modules behind. To avoid this, we use our own DaemonSet to remove the preinstalled driver. Another option is to recreate the node group without preinstalled drivers. After that, we can let GPU Operator manage the nodes without worrying about conflicts.

My other articles:

Building an ML Platform on Kubernetes: Yandex Cloud, JupyterHub, Dask and S3

The Nuances of Updating NVIDIA Drivers in Yandex Managed Kubernetes