This wraps `nvproxy`'s `ioctl` handlers with wrappers that are
driver-capability-aware, and initially populates the ABI tree
with all functions marked as being available for capabilities
`compute` and `utility`.
It is not yet possible for the user to specify which capability
they want to enable. This should not cause a regression for users
because the hardcoded effective capability set is the one that is
currently specified for all current handlers.
Updates #9452.
Updates #10856.
PiperOrigin-RevId: 697824737
Our current check of COS drivers often lags behind COS releases.
This is due to needing to preload GPU docker images onto the
images that run in our CI pipelines.
In addition, COS can be a bit more complex than originally thought
releasing driver versions both across GPU types and release branches.
Thus, this test searches the latest COS images on each family for
new drivers. It does this by looking at COS's published release notes
which include a proto of LATEST/DEFAULT drivers selected for each device.
This will flag new versions faster with more coverage than our
CI pipeline currently. Due to this not actually needing a GPU
to run, this can run on any VM.
PiperOrigin-RevId: 693736100
New test, before this CL:
```
Testing read/write syscalls on cudaMallocHost memory
cuda_malloc: write: Bad address
```
After this CL:
```
Testing read/write syscalls on cudaMallocHost memory
Testing cudaMallocManaged(flags=cudaMemAttachGlobal)
Testing cudaMallocManaged(flags=cudaMemAttachGlobal) with prefetching
Testing cudaMallocManaged(flags=cudaMemAttachHost)
Testing cudaMallocManaged(flags=cudaMemAttachHost) with prefetching
Testing read/write syscalls on cudaMallocManaged memory
All tests passed
```
Fixes#10879
PiperOrigin-RevId: 672721411
This is necessary to prevent osDescMem pinned page leaks when the osDescMem is
freed indirectly, e.g. by closing the `/dev/nvidiactl` FD that owns the
containing client.
- Switch from passing separate sentryIoctlParams to temporarily modifying
ioctlParams in-place, in order to ensure that callers of *Invoke functions
always observe ioctlParams updated by driver copy-out (including e.g. updated
handles).
- Only pass NVOS64Parameters (rather than NVOS21Parameters) in NV_ESC_RM_ALLOC
calls to the driver; this should behave equivalently (new comment in
rmAllocInvoke()) and simplifies passing them to object registration callbacks
(objAddLocked).
PiperOrigin-RevId: 627630863
Tested on a T4 GPU with driver version 525.60.13:
```
$ docker run --runtime=runsc --gpus=all --rm -it nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubi8
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
```
Also tested this on GKE with the same vectoradd workload. Checked that the
device gofer connection is actually being closed when the container is deleted.
Something to note is that the gofer logs for the GPU-container sometimes end
abruptly (the "All lisafs servers exited." line does not print). This is
because runsc/container/container.go:stop() SIGKILLs the gofer before it can
cleanup naturally. The device gofer connection is only closed at the end of
Loader.destroySubcontainer(), which gives little time before the gofer is
SIGKILL-ed.
PiperOrigin-RevId: 581365665
Our devtmpfs implementation uses the same tmpfs filesystem instance for all
devtmpfs mounts in the sandbox. This would mean that devices mounted in a
container are visible and accessible to all other containers in the sandbox.
With GPU/TPU, the contents of devtmpfs can be different for different
containers within the same sandbox. So it is important to not share the same
devtmpfs contents.
It is better to drop support for devtmpfs, than to implement it incorrectly.
Instead, this change introduces a new dummy filesystem type named `dev`. This
filesystem can not be mounted or listed by the application. This filesystem
creates a new tmpfs instance on GetFilesystem() and populates it with all the
device files.
PiperOrigin-RevId: 578969556
Nvproxy now supports 535.54.03 and 535.104.05 versions.
Tested on T4 GPU for these versions.
```
$ docker run --runtime=runsc --rm --gpus=all nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubi8
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
```
PiperOrigin-RevId: 572312006
This adds utilities to make it easier to add versioning support in nvproxy.
This way we can have different handling of ioctls based on driver versions.
The largest downside of this solution is the usage of maps instead of switch
statements. Switch statements are *way* faster than map accesses. The rationale
is that most of the latency bound work happens on the GPU for GPU workloads. So
micro-optimizations like using switch statements should not be visible.
Co-authored-by: Jamie Liu <jamieliu@google.com>
PiperOrigin-RevId: 570889581
Ayush RanjanandgVisor bot
Etienne PerotandgVisor bot
Zach KoopmansandgVisor bot
Jamie LiuandgVisor bot
Anthony CuiandgVisor bot
Jing ChenandgVisor bot