netbird/gvisor
0
0
Fork 0
mirror of https://github.com/netbirdio/gvisor.git synced 2026-05-22 17:12:49 -07:00
Code Issues Packages Projects Releases Wiki Activity

perf: Remove call to nvidia-container-cli info

Browse Source 
This call is repeated on every container startup sequence, but it is expensive, consistently taking between 2-3 seconds to run each time. This adds a significant amount of startup latency to every container. From `strace` output, the main issue appears to be a call to `openat(AT_FDCWD, "/dev/nvidia0", O_RDWR|O_CLOEXEC)`. The first time this is called, it creates a new open device file descriptor and blocks for 2 seconds while doing so. This was measured on a `g4dn.2xlarge` instance on AWS, running a Tesla T4 GPU.

For comparison, the `nvidia-container-prestart-hook` for runc also runs `nvidia-container-cli` during container boot, but it only calls it once for the `configure` command. gVisor appears to call it twice, once for `info` and then for `configure`.

By removing the `info` call, or at least only running it when the GPU device files are not already present on the host, GPU container startups can be 2-3 seconds faster.

Does this sound reasonable? I'm not actually familiar with why the `/dev/nvidia0` file takes 2-3 seconds to open, or why GPU device files need to be explicitly loaded. But I tested the change, and gVisor nvproxy still works.
This commit is contained in:
Eric Zhang
2023-09-11 19:51:18 +00:00
parent 9926c0f464
commit 8758e992c5
1 changed files with 21 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
runsc/container/container.go
+21 -15
View File
@@ -1700,22 +1700,28 @@ func nvProxyPreGoferHostSetup(spec *specs.Spec, conf *config.Config) error {
// nvidia-container-cli --load-kmods seems to be a noop; load kernel modules ourselves.
nvproxyLoadKernelModules()
// Run `nvidia-container-cli info`.
// This has the side-effect of automatically creating GPU device files.
argv := []string{cliPath, "--load-kmods", "info"}
log.Debugf("Executing %q", argv)
var infoOut, infoErr strings.Builder
cmd := exec.Cmd{
Path: argv[0],
Args: argv,
Env: os.Environ(),
Stdout: &infoOut,
Stderr: &infoErr,
if _, err := os.Stat("/dev/nvidiactl"); err != nil {
if !os.IsNotExist(err) {
return fmt.Errorf("stat(2) for /dev/nvidiactl failed: %w", err)
}
// Run `nvidia-container-cli info`.
// This has the side-effect of automatically creating GPU device files.
argv := []string{cliPath, "--load-kmods", "info"}
log.Debugf("Executing %q", argv)
var infoOut, infoErr strings.Builder
cmd := exec.Cmd{
Path: argv[0],
Args: argv,
Env: os.Environ(),
Stdout: &infoOut,
Stderr: &infoErr,
}
if err := cmd.Run(); err != nil {
return fmt.Errorf("nvidia-container-cli info failed, err: %v\nstdout: %s\nstderr: %s", err, infoOut.String(), infoErr.String())
}
log.Debugf("nvidia-container-cli info: %v", infoOut.String())
}
if err := cmd.Run(); err != nil {
return fmt.Errorf("nvidia-container-cli info failed, err: %v\nstdout: %s\nstderr: %s", err, infoOut.String(), infoErr.String())
}
log.Debugf("nvidia-container-cli info: %v", infoOut.String())
return nil
}