diff --git a/website/_config.yml b/website/_config.yml index 6097a4062..d7b34bdb1 100644 --- a/website/_config.yml +++ b/website/_config.yml @@ -25,6 +25,9 @@ defaults: layout: default analytics: "UA-150193582-1" authors: + bogomolov: + name: Konstantin Bogomolov + email: bogomolov@google.com lucasmanning: name: Lucas Manning email: lucasmanning@google.com diff --git a/website/assets/images/2023-04-28-getpid-ptrace-vs-native.svg b/website/assets/images/2023-04-28-getpid-ptrace-vs-native.svg new file mode 100644 index 000000000..84d8fbc3c --- /dev/null +++ b/website/assets/images/2023-04-28-getpid-ptrace-vs-native.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/website/assets/images/2023-04-28-getpid-ptrace-vs-systrap-opt.svg b/website/assets/images/2023-04-28-getpid-ptrace-vs-systrap-opt.svg new file mode 100644 index 000000000..95c749f6d --- /dev/null +++ b/website/assets/images/2023-04-28-getpid-ptrace-vs-systrap-opt.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/website/assets/images/2023-04-28-getpid-ptrace-vs-systrap-unopt.svg b/website/assets/images/2023-04-28-getpid-ptrace-vs-systrap-unopt.svg new file mode 100644 index 000000000..84d8fbc3c --- /dev/null +++ b/website/assets/images/2023-04-28-getpid-ptrace-vs-systrap-unopt.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/website/assets/images/2023-04-28-systrap-sample-workloads.svg b/website/assets/images/2023-04-28-systrap-sample-workloads.svg new file mode 100644 index 000000000..de179f24e --- /dev/null +++ b/website/assets/images/2023-04-28-systrap-sample-workloads.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/website/blog/2023-04-28-systrap-release.md b/website/blog/2023-04-28-systrap-release.md new file mode 100644 index 000000000..801f80a4a --- /dev/null +++ b/website/blog/2023-04-28-systrap-release.md @@ -0,0 +1,166 @@ +# Releasing Systrap - A high-performance gVisor platform + +We are releasing a new gVisor platform: Systrap. Like the existing ptrace +platform, Systrap runs on most Linux machines out of the box without +virtualization. Unlike the ptrace platform, it’s fast 🚀. Go try it by adding +`--platform=systrap` to the runsc flags. If you want to know more about it, read +on. + +-------------------------------------------------------------------------------- + +gVisor is a security boundary for arbitrary Linux processes. Boundaries do not +come for free, and gVisor imposes some performance overhead on sandboxed +applications. One of the most fundamental performance challenges with the +security model implemented by gVisor is system call interception, which is the +focus of this post. + +To recap on the +[security model](https://gvisor.dev/docs/architecture_guide/security/#what-can-a-sandbox-do): +gVisor is an application kernel that implements the Linux ABI. This includes +system calls, signals, memory management, and more. For example, when a +sandboxed application calls +[`read(2)`](https://man7.org/linux/man-pages/man2/read.2.html), it actually +transparently calls into +[gVisor's implementation of this system call](https://github.com/google/gvisor/blob/44e2d0fcfeb641f3b8013c3f93cacdae447cc0f1/pkg/sentry/syscalls/linux/sys_read_write.go#L36) +This minimizes the attack surface of the host kernel, because sandboxed programs +simply can’t make system calls directly to the host in the first place[^1]. This +interception happens through an internal layer called the Platform interface, +which we have written about in a previous +[blog post](https://gvisor.dev/blog/2020/10/22/platform-portability/). To handle +these interceptions, this interface must also create new address spaces, +allocate memory, and create execution contexts to run the workload. + +gVisor had two platform implementations: KVM and ptrace. The KVM platform uses +the kernel’s KVM functionality to allow the Sentry to act as both guest OS and +VMM (Virtual machine monitor). It does system call interception just like a +normal virtual machine would. This gives good performance when using bare-metal +virtualization, but has a noticeable impact with nested virtualization. The +other obvious downside is that it requires support for nested virtualization in +the first place, which is not supported by all hardware (such as ARM CPUs) or +within some Cloud environments. + +The ptrace platform was the alternative wherever KVM was not available. It works +through the +[`PTRACE_SYSEMU`](http://man7.org/linux/man-pages/man2/ptrace.2.html) action, +which makes the user process hand back execution to the sentry whenever it +encounters a system call. This is a clean method to achieve system call +interception in any environment, virtualized or not, except that it’s quite +slow. To see how slow, an unrealistic but highly illustrative benchmark to use +is the +[`getpid` benchmark](https://github.com/google/gvisor/blob/108410638aa8480e82933870ba8279133f543d2b/test/perf/linux/getpid_benchmark.cc)[^2]. +This benchmark runs the +[`getpid(2)`](https://man7.org/linux/man-pages/man2/getpid.2.html) system call +in a tight `while` loop. No useful application has this behavior, so it is not a +realistic benchmark, but it is well-suited to measure system call latency. + +![Figure 1](/assets/images/2023-04-28-getpid-ptrace-vs-native.svg "Getpid benchmark: ptrace vs. native Linux."){:width="100%"} + +All `getpid` runs have been performed on a GCE n2-standard-4 VM, with the +`debian-11-bullseye-v20230306` image. + +While this benchmark is not applicable to most real-world workloads, just about +any workload will generally suffer from high overhead in system call +performance. Since running in a virtualized environment is the default state for +most cloud users these days, it's important that gVisor performs well in this +context. Systrap is the new platform targeting this important use case. + +Systrap relies on multiple techniques to implement the Platform interface. Like +the ptrace platform, Systrap uses Linux's ptrace subsystem to initialize +workload executor threads, which are started as child processes of the main +gVisor sentry process. Systrap additionally sets a very restrictive seccomp +filter, installs a custom signal handler, and allocates chunks of memory shared +between user threads and runsc sentry. This shared memory is what serves as the +main form of communication between the sentry and sandboxed programs: whenever +the sandboxed process attempts to execute a system call, it triggers a `SIGSYS` +signal which is handled by our signal handler. The signal handler in turn +populates shared memory regions, and requests the sentry to handle the requested +system call. This alone proved to be faster than using `PTRACE_SYSEMU`, as +demonstrated by the `getpid` benchmark: + +![Figure 2](/assets/images/2023-04-28-getpid-ptrace-vs-systrap-unopt.svg "Getpid benchmark: ptrace vs. Systrap."){:width="100%"} + +Can we make it even faster? Recall what the main purpose of our signal handler +is: to send a request to the sentry via shared memory. To do that, the sandboxed +process must first incur the overhead of executing the seccomp filter[^3], and +then generating a full signal stack before being able to run the signal handler. +What if there was a way to simply have the sandboxed process jump to another +user-space function when it wanted to perform a system call? Well, turns out, +there is[^4] There is a popular x86 instruction pattern that’s used to perform +system calls, and it goes a little something like this: **`mov sysno, %eax; +syscall`**. The size of the mov instruction is 5 bytes and the size of the +syscall instruction is 2 bytes. Luckily this is just enough space to fit in a +**`jmp *%gs:offset`** instruction. When the signal handler sees this instruction +pattern, it signals to the sentry that the original instructions can be replaced +with a **`jmp`** to trampoline code that performs the same function as the +regular `SIGSYS` signal handler. The system call number is not lost, but rather +encoded in the offset. The results are even more impressive: + +![Figure 3](/assets/images/2023-04-28-getpid-ptrace-vs-systrap-opt.svg "Getpid benchmark: ptrace vs. Optimized Systrap."){:width="100%"} + +As mentioned, the `getpid` benchmark is not representative of real-world +performance. To get a better picture of the magnitude of improvement, here are +some real-world workloads: + +* The + [Build ABSL benchmark](https://github.com/google/gvisor/blob/master/blob/master/test/benchmarks/fs/bazel_test.go) + measures compilation performance by compiling + [abseil.io](https://abseil.io/); this is a highly system call dependent + workload due to needing to do a lot of I/O filesystem operations (gVisor’s + file system overhead is also dependent upon file system isolation it + implements, which is something you can learn about + [here](https://gvisor.dev/docs/user_guide/filesystem/)). +* The + [ffmpeg benchmark](https://github.com/google/gvisor/blob/master/blob/master/test/benchmarks/media/ffmpeg_test.go) + runs a multimedia processing tool, to perform video stream encoding/decoding + for example; this workload does not require a significant amount of system + calls and there are very few userspace to kernel mode switches. +* The + [Tensorflow benchmark](https://github.com/google/gvisor/blob/master/blob/master/test/benchmarks/ml/tensorflow_test.go) + trains a variety of machine learning models on CPU; the system-call usage of + this workload is in between compilation and ffmpeg, due to needing to + retrieve training and validation data, but the majority of time is still + spent just running userspace computations. +* Finally, the Redis benchmark performs SET RPC calls with 5 concurrent + clients, measures the latency that each call takes to execute, and reports + the median (scaled by 250,000 to fit the graph's axis); this workload is + heavily bounded by system call performance due to high network stack usage. + +![Figure 4](/assets/images/2023-04-28-systrap-sample-workloads.svg "Comparison of sample workloads running on ptrace, Systrap, and native Linux."){:width="100%"} + +Systrap will replace the ptrace platform by September 2023 and become the +default. Until then, we are working really hard to make it production-ready, +which includes working on additional performance and stability improvements, and +making sure we maintain a high bar for security through targeted fuzz-testing +for Systrap specifically. + +In the meantime, we would like gVisor users to try it out, and give us feedback! +If you run gVisor using ptrace today (either by specifying `--platform ptrace` +or not specifying the `--platform` flag at all), or you use the KVM platform with +nested virtualization, switching to Systrap should be a drop-in performance +upgrade. All you have to do is specify `--platform systrap` to runsc. If you +encounter any issues, please let us know at +[gvisor.dev/issues](https://github.com/google/gvisor/issues). +
+
+ +-------------------------------------------------------------------------------- + + + +[^1]: Even if the sandbox itself is compromised, it will still be bound by + several defense-in-depth layers, including a restricted set of seccomp + filters. You can find more details here: + [https://gvisor.dev/blog/2020/09/18/containing-a-real-vulnerability/](https://gvisor.dev/blog/2020/09/18/containing-a-real-vulnerability/). + +[^2]: Once the system call has been intercepted by gVisor (or in the case of + Linux, once the process has entered kernel-mode), actually executing the + getpid system call itself is very fast, so this benchmark effectively + measures single-thread syscall-interception overhead. + +[^3]: Seccomp filters are known to have a “not insubstantial” overhead: + [https://lwn.net/Articles/656307/](https://lwn.net/Articles/656307/). + +[^4]: On the x86_64 architecture, ARM does not have this optimization as of the + time of writing. + + diff --git a/website/blog/BUILD b/website/blog/BUILD index adc4d5d0e..d9afdf8f9 100644 --- a/website/blog/BUILD +++ b/website/blog/BUILD @@ -91,6 +91,16 @@ doc( permalink = "/blog/2022/10/24/buffer-pooling/", ) +doc( + name = "systrap_release", + src = "2023-04-28-systrap-release.md", + authors = [ + "bogomolov", + ], + layout = "post", + permalink = "/blog/2023/04/28/systrap-release/", +) + docs( name = "posts", deps = [