From 4cf25cc44e2a870a8c2b34edb6acbc8766736666 Mon Sep 17 00:00:00 2001
From: Ayush Ranjan <ayushranjan@google.com>
Date: Mon, 21 Oct 2024 14:02:24 -0700
Subject: [PATCH] Add instructions for configuring nvidia-container-runtime
 with containerd.

The only way to get containerd to use runsc shim correctly alongside
nvidia-container-runtime right now is to manually modify
/etc/containerd/config.toml after running `nvidia-ctk runtime configure`.

See https://github.com/google/gvisor/issues/10997#issuecomment-2423231931
and https://github.com/google/gvisor/issues/10997#issuecomment-2427611588.

Fixes #10997

PiperOrigin-RevId: 688264375
---
 g3doc/user_guide/containerd/configuration.md | 46 ++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/g3doc/user_guide/containerd/configuration.md b/g3doc/user_guide/containerd/configuration.md
index cbc339005..bb52389f5 100644
--- a/g3doc/user_guide/containerd/configuration.md
+++ b/g3doc/user_guide/containerd/configuration.md
@@ -100,3 +100,49 @@ log_level = "debug"
   debug-log = "/var/log/runsc/%ID%/gvisor.%COMMAND%.log"
 EOF
 ```
+
+## NVIDIA Container Runtime
+
+If you want to use
+[`nvidia-container-runtime`](https://developer.nvidia.com/container-runtime)
+with runsc through containerd, you might need to configure `nvidia` runtime in
+containerd via `sudo nvidia-ctk runtime configure --runtime=containerd` command.
+This will update `/etc/containerd/config.toml` with a new runtime named
+`nvidia`. However, this runtime's configuration is not compatible with runsc:
+
+-   Its `runtime_type` is set to runc. You will need to manually update this
+    field to specify runsc so that containerd tries to invoke
+    `containerd-shim-runsc-v1` when using `nvidia` runtime.
+-   Its `options` attempts to specify `BinaryName =
+    "/usr/bin/nvidia-container-runtime"`. However, runsc shim takes
+    configuration via `ConfigPath` as shown above. So the `options` needs to be
+    updated to specify `ConfigPath` and in the config.toml file needs to specify
+    the `BinaryName`.
+
+The `/etc/containerd/config.toml` file should look like:
+
+```
+...
+[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
+  runtime_type = "io.containerd.runsc.v1"
+
+[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
+  TypeUrl = "io.containerd.runsc.v1.options"
+  ConfigPath = "/etc/containerd/runsc.toml"
+...
+```
+
+And `/etc/containerd/runsc.toml` should look something like:
+
+```
+log_path = "/var/log/runsc/%ID%/shim.log"
+log_level = "debug"
+binary_name = "/usr/bin/nvidia-container-runtime"
+[runsc_config]
+  debug = "true"
+  debug-log = "/var/log/runsc/%ID%/gvisor.%COMMAND%.log"
+  nvproxy = "true"
+```
+
+See [this section](../gpu.md#nvidia-container-runtime) for information about
+configuring `nvidia-container-runtime` to use `runsc` as its low-level runtime.