Add image for ARM workloads for cuda-tests and mark tests that work on ARM.
Most tests don't work due to cross-compiling between sbma and aarch64.
However, a few do. Add an image to support them.
These CUDA tests were initially broken in gVisor but now appear to pass.
The test now also verifies that all capabilities are enabled when running.
PiperOrigin-RevId: 713094806
This refreshes the set of models built into the image to a more diverse
set of models while keeping the same categories covered.
It also adds support for embedding generation and benchmark metrics for
embedding-type models.
The image is also (slightly) smaller which helps make benchmarks not take
forever.
PiperOrigin-RevId: 706594537
Per https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#id5,
CUDA 12.3.0 requires driver `>=545.23.06`, while CUDA 12.2.2 requires driver
`>=535.104.05`. Per nvproxy/version.go, 535.104.05 is currently the oldest
driver version we support.
PiperOrigin-RevId: 694652445
New test, before this CL:
```
Testing read/write syscalls on cudaMallocHost memory
cuda_malloc: write: Bad address
```
After this CL:
```
Testing read/write syscalls on cudaMallocHost memory
Testing cudaMallocManaged(flags=cudaMemAttachGlobal)
Testing cudaMallocManaged(flags=cudaMemAttachGlobal) with prefetching
Testing cudaMallocManaged(flags=cudaMemAttachHost)
Testing cudaMallocManaged(flags=cudaMemAttachHost) with prefetching
Testing read/write syscalls on cudaMallocManaged memory
All tests passed
```
Fixes#10879
PiperOrigin-RevId: 672721411
The benchmark loads a small model
to allow running on many platforms
and to keep execution time low for
experimentation. It will make sense
to add a version to this test that
loads a larger model (e.g. llama3-8b).
This also adds two new flags, --enforce_compatibility and --verbose, and
fixes an issue where legacy control ioctls were being flagged as unsupported.
PiperOrigin-RevId: 645193498
New test, before this change:
```
Testing cudaMallocManaged(flags=cudaMemAttachGlobal)
Testing cudaMallocManaged(flags=cudaMemAttachGlobal) with prefetching
Testing cudaMallocManaged(flags=cudaMemAttachHost)
Testing cudaMallocManaged(flags=cudaMemAttachHost) with prefetching
Testing read/write syscalls on cudaMallocManaged memory
cuda_malloc_managed: write: Bad address
```
After this change:
```
Testing cudaMallocManaged(flags=cudaMemAttachGlobal)
Testing cudaMallocManaged(flags=cudaMemAttachGlobal) with prefetching
Testing cudaMallocManaged(flags=cudaMemAttachHost)
Testing cudaMallocManaged(flags=cudaMemAttachHost) with prefetching
Testing read/write syscalls on cudaMallocManaged memory
All tests passed
```
Fixes#10331
PiperOrigin-RevId: 629859064
This is a set of CUDA tests defined by NVIDIA in this repository:
https://github.com/NVIDIA/cuda-samples
This change introduces a large new test (`cuda_test`) which runs each CUDA
sample test in a container.
There are many subtleties involved due to how the CUDA samples repository
isn't always meant to be run as a test, some of it involves graphical
applications, and a lot of them require this or that CUDA feature which not
all NVIDIA GPUs support, some require multiple GPUs to be on the machine, etc.
Therefore, the test maps each test to their `Compatibility` data which
determines whether or not a sample test is expected to fail when run in a
certain environment. The overall test also has a
`--cuda_verify_compatibility` flag to verify the veracity of this mapping,
by running expected-to-be-broken tests and verifying that their failure
matches how this expected failure typically manifests.
Because there are a lot of CUDA sample tests (213 as of the CUDA 12.3 release
of the cuda-samples repo), and they don't all require the whole GPU to
themselves, and spawning a GPU-using container is expensive (~seconds),
the test uses a pool of reusable containers in which it `exec`s tests (at
most one per container at any given time, but in parallel across containers).
If any test unexpectedly fails, we drain the entire pool of containers and
only run this one test without anything else running on other containers.
This de-flakes tests, especially those that fail because they require more
resources than the GPU has when other tests are using it at the same time.
However, this removes parallelism and therefore increases test time
significantly.
Despite these optimizations, the test is very long and has the maximum deadline
of 1 hour. Because it may get close to the timeout (especially when
`--cuda_verify_compatibility` is on, because that needs to run all the tests
even if they are known to fail, and the ones that do fail can hang for a
while rather than crash), the test also has more logging and debugging than
the typical test, as enabled with the `--cuda_log_successful_tests` and
`--cuda_test_debug` flag. It periodically logs the status of each container
and the pool's utilization ratio. This is useful when debugging the test to
see which pooled container is doing what, and/or to `docker exec` into
containers while they are running a certain test.
The test also does its own timekeeping, which is useful so that it can print
a more helpful failure message that distinguishes between tests failing due
to actual failure reasons vs those that are failing purely because the test
timed out.
To run the test manually (from a VM with the repo checked out):
```
$ docker build -t gvisor.dev/images/gpu/cuda-tests images/gpu/cuda-tests -f images/gpu/cuda-tests/Dockerfile.x86_64 && mkdir -p bin && make copy TARGETS=runsc DESTINATION=bin/ && ./bin/runsc install -- --nvproxy=true --debug=true --debug-log=/tmp/runsc/ && systemctl reload docker && make test TEST_OPTIONS='--test_output=streamed --verbose_failures=true' TARGETS=//test/gpu:cuda_test OPTIONS='--test_env=RUNTIME=runsc --test_arg=--cuda_test_debug=true --test_arg=cuda_verify_compatibility=true --test_arg=--cuda_log_successful_tests=true'
```
PiperOrigin-RevId: 626528982
PIL requires a string format name, whereas `Format.PNG` is an "enum string"
which is not quite the same type.
I had not noticed this problem because I had manually tested the test on a VM
where the cached Stable Diffusion XL Docker image was from before this change.
Tested on a fresh VM with a freshly-built Docker image now.
This is split in two changes because the image needs to be rebuilt.
The first change updates the image, and the second change re-enables the test.
PiperOrigin-RevId: 618048266
Yes, it's yet another huge image (26GB) :)
This image is useful to benchmark checkpoint/restore performance.
To fairly measure it, we need an image that has only one model
preloaded into it, as this mirrors the situation where someone might
want to use checkpoint/restore for an inference model where there is
only one model to serve.
This image uses mixtral as a medium-large model. The larger the model,
the more important the gains in checkpoint/restore performance, so
measuring with a large model provides a more representative and
significant result.
PiperOrigin-RevId: 613335037
Zach KoopmansandgVisor bot
Ayush RanjanandgVisor bot
Etienne PerotandgVisor bot
Jamie LiuandgVisor bot
Anthony CuiandgVisor bot
Otto Bittner
Andrei VaginandgVisor bot