diff --git a/website/assets/images/2023-06-27-directfs.svg b/website/assets/images/2023-06-27-directfs.svg new file mode 100644 index 000000000..b2abd9fc8 --- /dev/null +++ b/website/assets/images/2023-06-27-directfs.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/website/assets/images/2023-06-27-gofer-proxy.svg b/website/assets/images/2023-06-27-gofer-proxy.svg new file mode 100644 index 000000000..25f34fdd9 --- /dev/null +++ b/website/assets/images/2023-06-27-gofer-proxy.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/website/assets/images/2023-06-27-real-world-benchmarks.svg b/website/assets/images/2023-06-27-real-world-benchmarks.svg new file mode 100644 index 000000000..763f5cdd8 --- /dev/null +++ b/website/assets/images/2023-06-27-real-world-benchmarks.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/website/assets/images/2023-06-27-stat-benchmark.svg b/website/assets/images/2023-06-27-stat-benchmark.svg new file mode 100644 index 000000000..ba4538dbb --- /dev/null +++ b/website/assets/images/2023-06-27-stat-benchmark.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/website/blog/2023-06-27-directfs.md b/website/blog/2023-06-27-directfs.md new file mode 100644 index 000000000..330f890eb --- /dev/null +++ b/website/blog/2023-06-27-directfs.md @@ -0,0 +1,145 @@ +# Faster filesystem access with Directfs + +Directfs is now the default in runsc. This feature gives gVisor’s application +kernel (the Sentry) secure direct access to the container filesystem, avoiding +expensive round trips to the filesystem gofer. Learn more about this feature in +the following blog that was +[originally posted](https://opensource.googleblog.com/2023/06/optimizing-gvisor-filesystems-with-directfs.html) +on [Google Open Source Blog](https://opensource.googleblog.com/). + +-------------------------------------------------------------------------------- + +## Origins of the Gofer + +gVisor is used internally at Google to run a variety of services and workloads. +One of the challenges we faced while building gVisor was providing remote +filesystem access securely to the sandbox. gVisor’s strict +[security model](https://gvisor.dev/docs/architecture_guide/security/) and +defense in depth approach assumes that the sandbox may get compromised because +it shares the same execution context as the untrusted application. Hence the +sandbox cannot be given sensitive keys and credentials to access Google-internal +remote filesystems. + +To address this challenge, we added a trusted filesystem proxy called a "gofer". +The gofer runs outside the sandbox, and provides a secure interface for +untrusted containers to access such remote filesystems. For architectural +simplicity, gofers were also used to serve local filesystems as well as remote. + +![Figure 1](/assets/images/2023-06-27-gofer-proxy.svg "Filesystem gofer proxy"){:width="100%"} + +## Isolating the Container Filesystem in runsc + +When gVisor was [open sourced](https://github.com/google/gvisor) as +[runsc](https://gvisor.dev/docs/), the same gofer model was copied over to +maintain the same security guarantees. runsc was configured to start one gofer +process per container which serves the container filesystem to the sandbox over +a predetermined protocol (now +[LISAFS](https://github.com/google/gvisor/blob/master/pkg/lisafs)). However, a gofer +adds a layer of indirection with significant overhead. + +This gofer model (built for remote filesystems) brings very few advantages for +the runsc use-case, where all the filesystems served by the gofer (like rootfs +and [bind mounts](https://docs.docker.com/storage/bind-mounts/)) are mounted +locally on the host. The gofer directly accesses them using filesystem syscalls. + +Linux provides some security primitives to effectively isolate local +filesystems. These include, +[mount namespaces](https://man7.org/linux/man-pages/man7/mount_namespaces.7.html), +[`pivot_root`](https://man7.org/linux/man-pages/man2/pivot_root.2.html) and +detached bind mounts[^1]. **Directfs** is a new filesystem access mode that uses +these primitives to expose the container filesystem to the sandbox in a secure +manner. The sandbox’s view of the filesystem tree is limited to just the +container filesystem. The sandbox process is not given access to anything +mounted on the broader host filesystem. Even if the sandbox gets compromised, +these mechanisms provide additional barriers to prevent broader system +compromise. + +## Directfs + +In directfs mode, the gofer still exists as a cooperative process outside the +sandbox. As usual, the gofer enters a new mount namespace, sets up appropriate +bind mounts to create the container filesystem in a new directory and then +[`pivot_root(2)`](https://man7.org/linux/man-pages/man2/pivot_root.2.html)s into +that directory. Similarly, the sandbox process enters new user and mount +namespaces and then +[`pivot_root(2)`](https://man7.org/linux/man-pages/man2/pivot_root.2.html)s into +an empty directory to ensure it cannot access anything via path traversal. But +instead of making RPCs to the gofer to access the container filesystem, the +sandbox requests the gofer to provide file descriptors to all the mount points +via [`SCM_RIGHTS` messages](https://man7.org/linux/man-pages/man7/unix.7.html). +The sandbox then directly makes file-descriptor-relative syscalls (e.g. +[`fstatat(2)`](https://linux.die.net/man/2/fstatat), +[`openat(2)`](https://linux.die.net/man/2/openat), +[`mkdirat(2)`](https://linux.die.net/man/2/mkdirat), etc) to perform filesystem +operations. + +![Figure 2](/assets/images/2023-06-27-directfs.svg "Directfs configuration"){:width="100%"} + +Earlier when the gofer performed all filesystem operations, we could deny all +these syscalls in the sandbox process using seccomp. But with directfs enabled, +the sandbox process's seccomp filters need to allow the usage of these syscalls. +Most notably, the sandbox can now make +[`openat(2)`](https://linux.die.net/man/2/openat) syscalls (which allow path +traversal), but with certain restrictions: +[`O_NOFOLLOW` is required](https://github.com/google/gvisor/commit/114a033bd038519fa6e867c230dc4ad4e057e675), +[no access to procfs](https://github.com/google/gvisor/commit/fcbc289a7ac14b8d84d0c0b23c4b2a14fc626e79) +and +[no directory FDs from the host](https://github.com/google/gvisor/commit/aa8abdfa9256cf057202ec8f4a81ba9f5d6a203f). +We also had to give the sandbox the same privileges as the gofer (for example +`CAP_DAC_OVERRIDE` and `CAP_DAC_READ_SEARCH`), so it can perform the same +filesystem operations. + +It is noteworthy that only the trusted gofer provides FDs (of the container +filesystem) to the sandbox. The sandbox cannot walk backwards (using '..') or +follow a malicious symlink to escape out of the container filesystem. In effect, +we've decreased our dependence on the syscall filters to catch bad behavior, but +correspondingly increased our dependence on Linux's filesystem isolation +protections. + +## Performance + +Making RPCs to the gofer for every filesystem operation adds a lot of overhead +to runsc. Hence, avoiding gofer round trips significantly improves performance. +Let's find out what this means for some of our benchmarks. We will run the +benchmarks using our newly released +[systrap platform](https://gvisor.dev/blog/2023/04/28/systrap-release/) on bind +mounts (as opposed to rootfs). This would simulate more realistic use cases +because bind mounts are extensively used while configuring filesystems in +containers. Bind mounts also do not have an overlay +([like the rootfs mount](https://opensource.googleblog.com/2023/04/gvisor-improves-performance-with-root-filesystem-overlay.html)), +so all operations go through goferfs / directfs mount. + +Let's first look at our +[stat micro-benchmark](https://github.com/google/gvisor/blob/master/test/perf/linux/stat_benchmark.cc), +which repeatedly calls +[`stat(2)`](https://man7.org/linux/man-pages/man2/lstat.2.html) on a file. + +![Figure 3](/assets/images/2023-06-27-stat-benchmark.svg "Stat micro benchmark"){:width="100%"} + +The `stat(2)` syscall is more than 2x faster! However, since this is not +representative of real-world applications, we should not extrapolate these +results. So let's look at some +[real-world benchmarks](https://github.com/google/gvisor/blob/master/test/benchmarks/fs). + +![Figure 4](/assets/images/2023-06-27-real-world-benchmarks.svg "Real world benchmarks"){:width="100%"} + +We see a 12% reduction in the absolute time to run these workloads and 17% +reduction in Ruby load time! + +## Conclusion + +The gofer model in runsc was overly restrictive for accessing host files. We +were able to leverage existing filesystem isolation mechanisms in Linux to +bypass the gofer without compromising security. Directfs significantly improves +performance for certain workloads. This is part of our ongoing efforts to +improve gVisor performance. You can learn more about gVisor at +[gvisor.dev](http://www.gvisor.dev/). You can also use gVisor in +[GKE](https://cloud.google.com/kubernetes-engine) with +[GKE Sandbox](https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods). +Happy sandboxing! + +-------------------------------------------------------------------------------- + +[^1]: Detached bind mounts can be created by first creating a bind mount using + mount(MS_BIND) and then detaching it from the filesystem tree using + umount(MNT_DETACH). diff --git a/website/blog/BUILD b/website/blog/BUILD index e437a37fe..bcc6694a0 100644 --- a/website/blog/BUILD +++ b/website/blog/BUILD @@ -121,6 +121,16 @@ doc( permalink = "/blog/2023/06/20/gpu-pytorch-stable-diffusion/", ) +doc( + name = "directfs", + src = "2023-06-27-directfs.md", + authors = [ + "ayushranjan", + ], + layout = "post", + permalink = "/blog/2023/06/27/directfs/", +) + docs( name = "posts", deps = [