mirror of
https://github.com/netbirdio/gvisor.git
synced 2026-05-22 17:12:49 -07:00
Etienne PerotandgVisor bot
4810afc36c
This is a set of CUDA tests defined by NVIDIA in this repository: https://github.com/NVIDIA/cuda-samples This change introduces a large new test (`cuda_test`) which runs each CUDA sample test in a container. There are many subtleties involved due to how the CUDA samples repository isn't always meant to be run as a test, some of it involves graphical applications, and a lot of them require this or that CUDA feature which not all NVIDIA GPUs support, some require multiple GPUs to be on the machine, etc. Therefore, the test maps each test to their `Compatibility` data which determines whether or not a sample test is expected to fail when run in a certain environment. The overall test also has a `--cuda_verify_compatibility` flag to verify the veracity of this mapping, by running expected-to-be-broken tests and verifying that their failure matches how this expected failure typically manifests. Because there are a lot of CUDA sample tests (213 as of the CUDA 12.3 release of the cuda-samples repo), and they don't all require the whole GPU to themselves, and spawning a GPU-using container is expensive (~seconds), the test uses a pool of reusable containers in which it `exec`s tests (at most one per container at any given time, but in parallel across containers). If any test unexpectedly fails, we drain the entire pool of containers and only run this one test without anything else running on other containers. This de-flakes tests, especially those that fail because they require more resources than the GPU has when other tests are using it at the same time. However, this removes parallelism and therefore increases test time significantly. Despite these optimizations, the test is very long and has the maximum deadline of 1 hour. Because it may get close to the timeout (especially when `--cuda_verify_compatibility` is on, because that needs to run all the tests even if they are known to fail, and the ones that do fail can hang for a while rather than crash), the test also has more logging and debugging than the typical test, as enabled with the `--cuda_log_successful_tests` and `--cuda_test_debug` flag. It periodically logs the status of each container and the pool's utilization ratio. This is useful when debugging the test to see which pooled container is doing what, and/or to `docker exec` into containers while they are running a certain test. The test also does its own timekeeping, which is useful so that it can print a more helpful failure message that distinguishes between tests failing due to actual failure reasons vs those that are failing purely because the test timed out. To run the test manually (from a VM with the repo checked out): ``` $ docker build -t gvisor.dev/images/gpu/cuda-tests images/gpu/cuda-tests -f images/gpu/cuda-tests/Dockerfile.x86_64 && mkdir -p bin && make copy TARGETS=runsc DESTINATION=bin/ && ./bin/runsc install -- --nvproxy=true --debug=true --debug-log=/tmp/runsc/ && systemctl reload docker && make test TEST_OPTIONS='--test_output=streamed --verbose_failures=true' TARGETS=//test/gpu:cuda_test OPTIONS='--test_env=RUNTIME=runsc --test_arg=--cuda_test_debug=true --test_arg=cuda_verify_compatibility=true --test_arg=--cuda_log_successful_tests=true' ``` PiperOrigin-RevId: 626528982
41 lines
1.5 KiB
Bash
41 lines
1.5 KiB
Bash
#!/bin/bash
|
|
|
|
# Copyright 2024 The gVisor Authors.
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
# This script outputs a sorted list of CUDA sample tests, one per line.
|
|
|
|
set -euo pipefail
|
|
|
|
(
|
|
while IFS= read -r makefile_path; do
|
|
dirname "$makefile_path"
|
|
done < <(find /cuda-samples -type f -name Makefile) \
|
|
| grep -vE '^/cuda-samples$' | grep -vE '/7_libNVVM'
|
|
|
|
# cuda-samples/Samples/7_libNVVM is not structured like the other tests.
|
|
# It is built with `cmake` and generates multiple test binaries.
|
|
# The generated ones all follow the pattern of being named after their
|
|
# parent directory name, so we look for that.
|
|
pushd /cuda-samples/Samples/7_libNVVM &>/dev/null
|
|
cmake . &>/dev/null
|
|
make TARGET_ARCH="$(uname -m)" all &>/dev/null
|
|
popd &>/dev/null
|
|
while IFS= read -r dir_path; do
|
|
if [[ -x "$dir_path/$(basename "$dir_path")" ]]; then
|
|
echo "$dir_path"
|
|
fi
|
|
done < <(find /cuda-samples/Samples/7_libNVVM -type d) | sort | uniq
|
|
) | sed 's~/cuda-samples/Samples/~~' | sort
|