diff --git a/website/_config.yml b/website/_config.yml index e394f39b4..6097a4062 100644 --- a/website/_config.yml +++ b/website/_config.yml @@ -25,6 +25,10 @@ defaults: layout: default analytics: "UA-150193582-1" authors: + lucasmanning: + name: Lucas Manning + email: lucasmanning@google.com + url: https://twitter.com/lucassmanning jsprad: name: Jeremiah Spradlin email: jsprad@google.com diff --git a/website/assets/images/2022-10-24-buffer-pooling-figure1.png b/website/assets/images/2022-10-24-buffer-pooling-figure1.png new file mode 100644 index 000000000..6688cb0a7 Binary files /dev/null and b/website/assets/images/2022-10-24-buffer-pooling-figure1.png differ diff --git a/website/assets/images/2022-10-24-buffer-pooling-figure2.png b/website/assets/images/2022-10-24-buffer-pooling-figure2.png new file mode 100644 index 000000000..e0c676c3e Binary files /dev/null and b/website/assets/images/2022-10-24-buffer-pooling-figure2.png differ diff --git a/website/blog/2022-10-24-buffer-pooling.md b/website/blog/2022-10-24-buffer-pooling.md new file mode 100644 index 000000000..32663d358 --- /dev/null +++ b/website/blog/2022-10-24-buffer-pooling.md @@ -0,0 +1,179 @@ +\# How we Eliminated 99% of gVisor Networking Memory Allocations with Enhanced +Buffer Pooling + +In an +[earlier blog post](https://gvisor.dev/blog/2020/04/02/gvisor-networking-security/) +about networking security, we described how and why gVisor implements its own +userspace network stack in the Sentry (gVisor kernel). In summary, we’ve +implemented our networking stack – aka Netstack – in Go to minimize exposure to +unsafe code and avoid using an unsafe Foreign Function Interface. With Netstack, +gVisor can do all packet processing internally and only has to enable a few host +I/O syscalls for near-complete networking capabilities. This keeps gVisor’s +exposure to host vulnerabilities as narrow as possible. + +Although writing Netstack in Go was important for runtime safety, up until now +it had an undeniable performance cost. iperf benchmarks showed Netstack was +spending between 20-30% of its processing time allocating memory and pausing for +garbage collection, a slowdown that limited gVisor’s ability to efficiently +sandbox networking workloads. In this blog we will show how we crafted a cure +for Netstack’s allocation addiction, reducing them by 99%, while also increasing +gVisor networking throughput by 30+%. + +![Figure 1](/assets/images/2022-10-24-buffer-pooling-figure1.png "Buffer pooling results.") + +## A Waste Management Problem + +Go guarantees a basic level of memory safety through the use of a garbage +collector (GC), which is described in great detail by the Go team +[here](https://tip.golang.org/doc/gc-guide). The Go runtime automatically tracks +and frees objects allocated from the heap, relieving the programmer of the often +painful and error-prone process of manual memory management. Unfortunately, +tracking and freeing memory during runtime comes at a performance cost. Running +the GC adds scheduling overhead, consumes valuable CPU time, and occasionally +pauses the entire program’s progress to track down garbage. + +Go’s GC is highly optimized, tunable, and sufficient for a majority of +workloads. Most of the other parts of gVisor happily use Go's GC with no +complaints. However, under high network stress, Netstack needed to aggressively +allocate buffers used for processing TCP/IP data and metadata. These buffers +often had short lifespans, and once the processing was done they were left to be +cleaned up by the GC. This meant Netstack was producing tons of garbage that +needed to be tracked and freed by GC workers. + +## Recycling to the Rescue + +Luckily, we weren't the only ones with this problem. This pattern of small, +frequently allocated and discarded objects was common enough that the Go team +introduced [`sync.Pool`](https://pkg.go.dev/sync#Pool) in Go1.3. `sync.Pool` is +designed to relieve pressure off the Go GC by maintaining a thread-safe cache of +previously allocated objects. `sync.Pool` can retrieve an object from the cache +if it exists or allocate a new one according to a user specified allocation +function. Once the user is finished with an object they can safely return it to +the cache to be reused again. + +While `sync.Pool` was exactly what we needed to reduce allocations, +incorporating it into Netstack wasn’t going to be as easy as just replacing all +our `make()`s with `pool.Get()`s. + +## Netstack Challenges + +Netstack uses a few different types of buffers under the hood. Some of these are +specific to protocols, like +[`segment`](https://github.com/google/gvisor/blob/master/pkg/tcpip/transport/tcp/segment.go) +for TCP, and others are more widely shared, like +[`PacketBuffer`](https://github.com/google/gvisor/blob/master/pkg/tcpip/stack/packet_buffer.go), +which is used for IP, ICMP, UDP, etc. Although each of these buffer types are +slightly different, they generally share a few common traits that made it +difficult to use `sync.Pool` out of the box: + +* The buffers were originally built with the assumption that a garbage + collector would clean them up automatically – there was little (if any) + effort put into tracking object lifetimes. This meant that we had no way to + know when it was safe to return buffers to a pool. +* Buffers have dynamic sizes that are determined during creation, usually + depending on the size of the packet holding them. A `sync.Pool` out of the + box can only accommodate buffers of a single size. One common solution to + this is to fill a pool with + [`bytes.Buffer`](https://pkg.go.dev/bytes#Buffer), but even a pooled + `bytes.Buffer` could incur allocations if it were too small and had to be + grown to the requested size. +* Netstack splits, merges, and clones buffers at various points during + processing (for example, breaking a large segment into smaller MTU-sized + packets). Modifying a buffer’s size during runtime could mean lots of + reallocating from the pool in a one-size-fits-all setup. This would limit + the theoretical effectiveness of a pooled solution. + +We needed an efficient, low-level buffer abstraction that had answers for the +Netstack specific challenges and could be shared by the various intermediate +buffer types. By sharing a common buffer abstraction, we could maximize the +benefits of pooling and avoid introducing additional allocations while minimally +changing any intermediate buffer processing logic. + +## Introducing bufferv2 + +Our solution was +[bufferv2](https://github.com/google/gvisor/tree/master/pkg/bufferv2). Bufferv2 +is a non-contiguous, reference counted, pooled, copy-on-write, buffer-like data +structure. + +Internally, a bufferv2 `Buffer` is a linked list of `View`s. Each `View` has +start/end indices and holds a pointer to a `Chunk`. A `Chunk` is a +reference-counted structure that’s allocated from a pool and holds data in a +byte slice. There are several `Chunk` pools, each of which allocates chunks with +different sized byte slices. These sizes start at 64 and double until 64k. + +![Figure 2](/assets/images/2022-10-24-buffer-pooling-figure2.png "bufferv2 implementation diagram.") + +The design of bufferv2 has a few key advantages over simpler object pooling: + +* **Zero-cost copies and copy-on-write**: Cloning a Buffer only increments the + reference count of the underlying chunks instead of reallocating from the + pool. Since buffers are much more frequently read than modified, this saves + allocations. In the cases where a buffer is modified, only the chunk that’s + changed has to be cloned, not the whole buffer. +* **Fast buffer transformations**: Truncating and merging buffers or appending + and prepending Views to Buffers are fast operations. Thanks to the + non-contiguous memory structure these operations are usually as quick as + adding a node to a linked list or changing the indices in a View. +* **Tiered pools**: When growing a Buffer or appending data, the new chunks + come from different pools of previously allocated chunks. Using multiple + pools means we are flexible enough to efficiently accommodate packets of all + sizes with minimal overhead. Unlike a one-size-fits-all solution, we don't + have to waste lots of space with a chunk size that is too big or loop + forever allocating small chunks. + +## Trade-offs + +Shifting Netstack to bufferv2 came with some costs. To start, rewriting all +buffers to use bufferv2 was a sizable effort that took many months to fully roll +out. Any place in Netstack that allocated or used a byte slice needed to be +rewritten. Reference counting had to be introduced so all the aforementioned +intermediate buffer types (`PacketBuffer`, `segment`, etc) could accurately +track buffer lifetimes, and tests had to be modified to ensure reference +counting correctness. + +In addition to the upfront cost, the shift to bufferv2 also increased the +engineering complexity of future Netstack changes. Netstack contributors must +adhere to new rules to maintain memory safety and maximize the benefits of +pooling. These rules are strict – there needs to be strong justification to +break them. They are as follows: + +* Never allocate a byte slice; always use `NewView()` instead. +* Use a `View` for simple data operations (e.g writing some data of a fixed + size) and a `Buffer` for more complex I/O operations (e.g appending data of + variable size, merging data, writing from an `io.Reader`). +* If you need access to the contents of a `View` as a byte slice, use + `View.AsSlice()`. If you need access to the contents of a `Buffer` as a byte + slice, consider refactoring, as this will cause an allocation. +* Never write or modify the slices returned by `View.AsSlice()`; they are + still owned by the view. +* Release bufferv2 objects as close to where they're created as possible. This + is usually most easily done with defer. +* Document function ownership of bufferv2 object parameters. If there is no + documentation, it is assumed that the function does not take ownership of + its parameters. +* If a function takes ownership of its bufferv2 parameters, the bufferv2 + objects must be cloned before passing them as arguments. +* All new Netstack tests must enable the leak checker and run a final leak + check after the test is complete. + +## Give it a Try + +Bufferv2 is enabled by default as of +[gVisor 20221017](https://github.com/google/gvisor/releases/tag/release-20221017.0), +and will be rolling out to +[GKE Sandbox](https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods) +soon, so no action is required to see a performance boost. Network-bound +workloads, such as web servers or databases like Redis, are the most likely to +see benefits. All the code implementing bufferv2 is public +[here](https://github.com/google/gvisor/tree/master/pkg/bufferv2), and +contributions are welcome! If you’d like to run the iperf benchmark for +yourself, you can run: + +``` +make run-benchmark BENCHMARKS_TARGETS=//test/benchmarks/network:iperf_test \ + RUNTIME=your-runtime-here BENCHMARKS_OPTIONS=-test.benchtime=60s +``` + +in the base gVisor directory. If you experience any issues, please feel free to +let us know at [gvisor.dev/issues](https://github.com/google/gvisor/issues). diff --git a/website/blog/BUILD b/website/blog/BUILD index 7c1fed439..65fcefbb8 100644 --- a/website/blog/BUILD +++ b/website/blog/BUILD @@ -80,6 +80,16 @@ doc( permalink = "/blog/2022/08/01/threat-detection/", ) +doc( + name = "buffer_pooling", + src = "2022-10-24-buffer-pooling.md", + authors = [ + "lucasmanning", + ], + layout = "post", + permalink = "/blog/2022/10/24/buffer-pooling/", +) + docs( name = "posts", deps = [