diff --git a/g3doc/user_guide/tpu.md b/g3doc/user_guide/tpu.md new file mode 100644 index 000000000..67e7fd41e --- /dev/null +++ b/g3doc/user_guide/tpu.md @@ -0,0 +1,142 @@ +# TPU Support + +[TOC] + +gVisor can add a layer of security to your TPU-based applications. By running +these applications in a sandboxed environment, you can isolate your host system +from potential vulnerabilities in code. This is crucial for handling sensitive +data or deploying untrusted workloads. + +gVisor supports running workloads that leverage TPU accelerators by proxying +hardware driver commands to the host with a feature called `tpuproxy`. +`tpuproxy` exposes host TPU devices to the sandbox so users can run their TPU +applications without any modifications. + +## Enabling + +The `runsc` flag `--tpuproxy` must be specified to enable TPU support. In GKE +this is done automatically for any sandbox node using a supported TPU machine +type. + +## Compatibility + +gVisor supports a wide range of workloads, including PyTorch and various +generative models like LLMs. Check out +[this blog post about running Stable Diffusion with gVisor](/blog/2023/06/20/gpu-pytorch-stable-diffusion/). +gVisor undergoes continuous tests to ensure this functionality remains robust. + +`tpuproxy` is a passthrough driver that forwards `ioctl(2)` calls made to TPU +devices by the containerized application directly to the host TPU driver. This +forwarding is straightforward: `ioctl` parameters are copied from the +application's address space to the sentry's address space, and then a host +`ioctl` syscall is made. `ioctl`s are passed through with minimal intervention. +`tpuproxy` also sets up a proxy sysfs filesystem that enables reading +configuration and status information of TPU devices on the host PCI bus. This +design translates to minimal overhead and maximal compatibility for TPU +operations, ensuring that TPU bound workloads experience negligible performance +impact. + +### Supported TPUs {#tpu-models} + +gVisor currently supports TPU models: V4lite V4pod, V5, and V5e. +[open a GitHub issue](https://github.com/google/gvisor/issues/new?labels=type%3A+enhancement,area%3A+gpu&template=bug_report.yml) +if you want support for another TPU model. gVisor only supports "1VM" TPU +shapes. + +### Supported Device Files {#device-files} + +gVisor exposes `/dev/accel[0-9]+` for TPU V4 and below. For TPU V5 and beyond, +gVisor exposes `/dev/vfio` and `/dev/vfio/[0-9]+`. For all versions, `tpuproxy` +exposes a read-only copy of the contents of TPU PCI device files located in the +host's sysfs directory. + +## Security + +Although `tpuproxy` enables sandboxed applications to run TPU accelerated +workloads, it does not provide the same level of isolation from host hardware +that it does for traditional CPU workloads. At a high level this is because +gVisor emulates the Linux kernel, which itself has limited control over the +memory isolation and compute scheduling of external devices. A more detailed +discussion follows: + +First, a short overview of +[gvisor's security model](../architecture_guide/security.md). gVisor protects +the host from sandboxed applications by providing several layers of defense. The +layers most relevant to this discussion are the redirection of application +syscalls to the gVisor sandbox and use of +[seccomp-bpf](https://www.kernel.org/doc/html/v4.19/userspace-api/seccomp_filter.html) +on gVisor sandboxes. + +gVisor uses a "platform" to tell the host kernel to reroute system calls to the +sandbox process, known as the sentry. The sentry implements a syscall table, +which services all application syscalls. The Sentry *may* make syscalls to the +host kernel if it needs them to fulfill the application syscall, but it doesn't +merely pass an application syscall to the host kernel. + +On sandbox boot, seccomp filters are applied to the sandbox. Seccomp filters +applied to the sandbox constrain the set of syscalls that it can make to the +host kernel, blocking access to most host kernel vulnerabilities even if the +sandbox becomes compromised. + +For example, [CVE-2022-0185](https://nvd.nist.gov/vuln/detail/CVE-2022-0185) is +mitigated because gVisor itself handles the syscalls required to use namespaces +and capabilities, so the application is using gVisor's implementation, not the +host kernel's. For a compromised sandbox, the syscalls required to exploit the +vulnerability are blocked by seccomp filters. + +In addition, seccomp-bpf filters can filter by argument names allowing us to +allowlist granularly by `ioctl(2)` arguments. `ioctl(2)` is a source of many +bugs in any kernel due to the complexity of its implementation. As of writing, +gVisor does +[allowlist some `ioctl`s](https://github.com/google/gvisor/blob/ccc3c2cbd26d3514885bd665b0a110150a6e8c53/runsc/boot/filter/config/config_main.go#L111) +by argument for things like terminal support. + +For example, [CVE-2024-21626](https://nvd.nist.gov/vuln/detail/CVE-2024-21626) +is mitigated by gVisor because the application would use gVisor's implementation +of `ioctl(2)`. For a compromised sentry, `ioctl(2)` calls with the needed +arguments are not in the seccomp filter allowlist, blocking the attacker from +making the call. gVisor also mitigates similar vulnerabilities that come with +device drivers +([CVE-2023-33107](https://nvd.nist.gov/vuln/detail/CVE-2023-33107)). + +### tpuproxy Security + +Recall that `tpuproxy` allows applications to directly interact with supported +ioctls used by the TPU driver. + +gVisor's seccomp filter rules are modified such that `ioctl(2)` calls can be +made +[*only for supported ioctls*](https://github.com/google/gvisor/blob/be9169a6ce095a08b99940a97db3f58e5c5bd2ce/pkg/sentry/devices/accel/seccomp_filters.go). +This approach is similar to the allowlisted ioctls for terminal support +described above. This allows gVisor to retain the vast majority of its +protection for the host while allowing access to TPUs. All of the above CVEs +remain mitigated even when `tpuproxy` is used. + +However, gVisor is much less effective at mitigating vulnerabilities within the +TPU drivers themselves, *because* gVisor passes through calls to be handled by +the kernel driver. If there is a vulnerability in the TPU driver for a given +`ioctl` that gVisor passes through, then gVisor will also be vulnerable. + +In addition, gVisor doesn't introduce any additional hardware-level isolation +beyond that which is configured by the host. There is no validation of things +like DMA buffers. The only checks are done in seccomp-bpf rules to ensure +`ioctl(2)` calls are made on supported and allowlisted `ioctl`s. + +NOTE: TPU V5 and beyond uses the +[VFIO Linux interface](https://docs.kernel.org/driver-api/vfio.html) to drive +TPU hardware. Theoretically VFIO could be used to configure memory isolation +using the host IOMMU. However, this requires manual setup by the user +application and does not come configured out of the box by gVisor. + +### So, if you don't protect against all the things, why even? + +While gVisor doesn't protect against *all* TPU driver vulnerabilities, it *does* +protect against a large set of general vulnerabilities in Linux. Applications +don't just use TPUs, they use them as a part of a larger application that may +include third party libraries. For example, Tensorflow +[suffers from the same kind of vulnerabilities](https://nvd.nist.gov/vuln/detail/CVE-2022-29216) +that every application does. Designing and implementing an application with +security in mind is hard and in the emerging AI space, security is often +overlooked in favor of getting to market fast. There are also many services that +allow users to run external users' code on the vendor's infrastructure. gVisor +is well suited as part of a larger security plan for these and other use cases.