Add image for ARM workloads for cuda-tests and mark tests that work on ARM.
Most tests don't work due to cross-compiling between sbma and aarch64.
However, a few do. Add an image to support them.
These CUDA tests were initially broken in gVisor but now appear to pass.
The test now also verifies that all capabilities are enabled when running.
PiperOrigin-RevId: 713094806
Per https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#id5,
CUDA 12.3.0 requires driver `>=545.23.06`, while CUDA 12.2.2 requires driver
`>=535.104.05`. Per nvproxy/version.go, 535.104.05 is currently the oldest
driver version we support.
PiperOrigin-RevId: 694652445
New test, before this CL:
```
Testing read/write syscalls on cudaMallocHost memory
cuda_malloc: write: Bad address
```
After this CL:
```
Testing read/write syscalls on cudaMallocHost memory
Testing cudaMallocManaged(flags=cudaMemAttachGlobal)
Testing cudaMallocManaged(flags=cudaMemAttachGlobal) with prefetching
Testing cudaMallocManaged(flags=cudaMemAttachHost)
Testing cudaMallocManaged(flags=cudaMemAttachHost) with prefetching
Testing read/write syscalls on cudaMallocManaged memory
All tests passed
```
Fixes#10879
PiperOrigin-RevId: 672721411
This also adds two new flags, --enforce_compatibility and --verbose, and
fixes an issue where legacy control ioctls were being flagged as unsupported.
PiperOrigin-RevId: 645193498
New test, before this change:
```
Testing cudaMallocManaged(flags=cudaMemAttachGlobal)
Testing cudaMallocManaged(flags=cudaMemAttachGlobal) with prefetching
Testing cudaMallocManaged(flags=cudaMemAttachHost)
Testing cudaMallocManaged(flags=cudaMemAttachHost) with prefetching
Testing read/write syscalls on cudaMallocManaged memory
cuda_malloc_managed: write: Bad address
```
After this change:
```
Testing cudaMallocManaged(flags=cudaMemAttachGlobal)
Testing cudaMallocManaged(flags=cudaMemAttachGlobal) with prefetching
Testing cudaMallocManaged(flags=cudaMemAttachHost)
Testing cudaMallocManaged(flags=cudaMemAttachHost) with prefetching
Testing read/write syscalls on cudaMallocManaged memory
All tests passed
```
Fixes#10331
PiperOrigin-RevId: 629859064
This is a set of CUDA tests defined by NVIDIA in this repository:
https://github.com/NVIDIA/cuda-samples
This change introduces a large new test (`cuda_test`) which runs each CUDA
sample test in a container.
There are many subtleties involved due to how the CUDA samples repository
isn't always meant to be run as a test, some of it involves graphical
applications, and a lot of them require this or that CUDA feature which not
all NVIDIA GPUs support, some require multiple GPUs to be on the machine, etc.
Therefore, the test maps each test to their `Compatibility` data which
determines whether or not a sample test is expected to fail when run in a
certain environment. The overall test also has a
`--cuda_verify_compatibility` flag to verify the veracity of this mapping,
by running expected-to-be-broken tests and verifying that their failure
matches how this expected failure typically manifests.
Because there are a lot of CUDA sample tests (213 as of the CUDA 12.3 release
of the cuda-samples repo), and they don't all require the whole GPU to
themselves, and spawning a GPU-using container is expensive (~seconds),
the test uses a pool of reusable containers in which it `exec`s tests (at
most one per container at any given time, but in parallel across containers).
If any test unexpectedly fails, we drain the entire pool of containers and
only run this one test without anything else running on other containers.
This de-flakes tests, especially those that fail because they require more
resources than the GPU has when other tests are using it at the same time.
However, this removes parallelism and therefore increases test time
significantly.
Despite these optimizations, the test is very long and has the maximum deadline
of 1 hour. Because it may get close to the timeout (especially when
`--cuda_verify_compatibility` is on, because that needs to run all the tests
even if they are known to fail, and the ones that do fail can hang for a
while rather than crash), the test also has more logging and debugging than
the typical test, as enabled with the `--cuda_log_successful_tests` and
`--cuda_test_debug` flag. It periodically logs the status of each container
and the pool's utilization ratio. This is useful when debugging the test to
see which pooled container is doing what, and/or to `docker exec` into
containers while they are running a certain test.
The test also does its own timekeeping, which is useful so that it can print
a more helpful failure message that distinguishes between tests failing due
to actual failure reasons vs those that are failing purely because the test
timed out.
To run the test manually (from a VM with the repo checked out):
```
$ docker build -t gvisor.dev/images/gpu/cuda-tests images/gpu/cuda-tests -f images/gpu/cuda-tests/Dockerfile.x86_64 && mkdir -p bin && make copy TARGETS=runsc DESTINATION=bin/ && ./bin/runsc install -- --nvproxy=true --debug=true --debug-log=/tmp/runsc/ && systemctl reload docker && make test TEST_OPTIONS='--test_output=streamed --verbose_failures=true' TARGETS=//test/gpu:cuda_test OPTIONS='--test_env=RUNTIME=runsc --test_arg=--cuda_test_debug=true --test_arg=cuda_verify_compatibility=true --test_arg=--cuda_log_successful_tests=true'
```
PiperOrigin-RevId: 626528982
This runs https://ollama.ai/ in a gVisor container and loads two models:
an English-Chinese translation model, and a code assistant model.
It asks the first one to translate "Hello World" to Chinese, and then asks
the second one to generate a test case to verify that the translation is
correct.
This change includes a server and client library for spawning ollama in a
container and interacting through its HTTP API. This will be useful to turn
it into a benchmark that measures its throughput in tokens/second.
PiperOrigin-RevId: 590295278
- The app attempts to allocate a driver object of class NV_CONFIDENTIAL_COMPUTE
in drivers too old to support said class; said allocation must fail with
status NV_ERR_INVALID_CLASS rather than errno EINVAL for the app to proceed.
- UVM_VALIDATE_VA_RANGE checks that a given address range is known to
nvidia-uvm. For mmaps of /dev/nvidia-uvm, this requires that the application
mmap (handled by nvproxy) immediately result in a host mmap (handled by the
driver). Ensure that mappings of nvproxy's nvidia-uvm have this property.
- Pass through UVM ioctl UVM_DISABLE_READ_DUPLICATION.
Fixes#9593
PiperOrigin-RevId: 577966817
Zach KoopmansandgVisor bot
Ayush RanjanandgVisor bot
Etienne PerotandgVisor bot
Jamie LiuandgVisor bot
Anthony CuiandgVisor bot
Andrei VaginandgVisor bot