Unlike Ubuntu VMs where we use Docker's `--gpus` flag, COS VMs do not use
this flag and instead mount the NVIDIA library directories automatically.
However, nothing guarantees that these directories are added to the LD
config. This change fixes that. It take advantage of the fact that all GPU
tests have the sniffer binary as entrypoint, which slightly overloads the
role of the sniffer within the GPU test infrastructure... but then again
the ioctl sniffer is already deeply intertwined with ld configuration
because it already overrides the `ioctl` libc function, so this doesn't
seem like too big of a stretch.
This change makes the ffmpeg test succeed with `runc` on COS, but they still
fail with gVisor (with `CUDA_ERROR_OUT_OF_MEMORY` errors). So there must be
some further gVisor-specific error.
Updates #11351
Updates #11321
PiperOrigin-RevId: 715222952
Before this change, there were 2 places in which the driver struct names were
defined for nvproxy structs:
1. As struct field tags. The first field of structs had a tag `nvproxy:*`. This
was kind of awkward. Such metadata is usually a struct comment.
2. In version.go while registering the struct with a name. Not all structs are
defined in nvproxy (for example simple structs). In such cases, the driver
struct name is directly assigned while registering struct info.
This change gets rid of (1). Most of the struct tags were `nvproxy:"same"`. Now
driverStructs() always infers the driver struct name using the nvproxy struct
name itself. The few cases where the nvproxy tag was needed, because driver
struct name was lower cased, were handled by defining driverStructWithName()
which allows specifying a different name. Now all driver struct names
definitions are in one place.
Along the way, also made the following fixes:
- For some reason, many structs defined in pkg/abi/nvgpu/frontend.go had
camel-cased naming, while all other structs in pkg/abi/nvgpu/ctrl.go and
pkg/abi/nvgpu/classes.go were named the same as their driver structs. The
convention in the abi/* packages is to follow the kernel source naming.
This is against Go sytle guide, but is more readable for gVisor purposes and
has been a long accepted convention. This also makes the task of removing (1)
easier. So renamed all such structs as per their driver names.
- A lot of code in pkg/sentry/devices/nvproxy/version.go was still referring to
driver struct info as "struct names", even though it was representing more
than just struct names. It also contains the reflect.Type of the struct which
is used to compare the nvproxy struct layout to the driver struct layout.
PiperOrigin-RevId: 710648105
This makes it easier to deal with client/server-type GPU tests, such as
ollama or vLLM, where the main GPU process is a long-running one. Prior to
this CL, when setting enforcement mode on this process, this would only be
checked if adding explicit code to shut down the container orderly by
sending a signal to the server process, in order to let the ioctl sniffer
have a chance to produce its report. This is easy to miss, because there
is no feedback that would suggest that the ioctl sniffer isn't being
respected in such tests. By exiting instantly as soon as an unsupported
ioctl is found, such code doesn't need to be added.
However, the previous behavior is still useful when testing new
applications, so it is still available as well. The
`--enforce_compatibility` flag is changed to a tri-state flag that can
handle either being turned off, or enforcing compatibility on the spot vs
at exit time.
Updates issue #10885.
PiperOrigin-RevId: 686653876
This wraps all GPU tests' command line with the nvproxy ioctl sniffer.
This has multiple functions:
- Verifying that the application does not call ioctls unsupported by
nvproxy. This is controlled by a `AllowIncompatibleIoctl` option, which
is initially set to `true` in all tests to mirror current behavior, but
should be flipped as we verify that they do not call unsupported ioctls.
- Verifying that the sniffer itself works transparently for a wide range
of applications.
- Later down the line, enforcing that the application only calls ioctls
that are part of GPU capabilities that it has a need for. This is
controlled by a capability string which is currently only used to set
the `NVIDIA_DRIVER_CAPABILITIES` environment variable.
Updates issue #10856
PiperOrigin-RevId: 672714520
Previously, run_sniffer was not correctly reporting when unsupported ioctls
were found with the compatibility flag set. At the same time, the sniffer test
was not correctly testing a supported CUDA program, since it was using
run_sample which is currently broken with the sniffer.
This also adds the sniffer test to the list of gpu tests.
PiperOrigin-RevId: 663911778
- This simplifies the code and makes it easier to add new ioctl classes in the
future. Results now just has a list of maps. Earlier, we had 2 map fields and
a field which was a map of map.
- Fixed some bugs in Ioctl.String() and enhanced the output.
PiperOrigin-RevId: 645499496
This also adds two new flags, --enforce_compatibility and --verbose, and
fixes an issue where legacy control ioctls were being flagged as unsupported.
PiperOrigin-RevId: 645193498
This makes it more convenient to use the `run_sniffer` tool as it is now
self-contained.
This also removes the use of RELR relocations for `libioctl_hook.so` as
I've found that they don't work on the `nvidia-smi` binary I'm working with.
This improves compatibility in general.
PiperOrigin-RevId: 644493298
Ayush RanjanandgVisor bot
Etienne PerotandgVisor bot
Anthony CuiandgVisor bot