gvisor.dev: Add Production Guide page.

This page does not document anything new, but it aims to be a central document to list all the steps and considerations necessary to run gVisor in a production environment. This includes: - How to make the decision to use sandboxing at all or not (which security benefits sandboxing bring, how to think about them). - How to make the decision about which components to sandbox. - What the security/performance tradeoffs are. - How to minimize the performance impact. - How to configure gVisor for best performance. Also includes a crude comparative diagrams of various ways to organize a typical production stack. PiperOrigin-RevId: 466491023
2026-05-22 17:12:49 -07:00 · 2022-08-09 15:47:11 -07:00
parent a963196f43
commit 241fd5344f
14 changed files with 2787 additions and 28 deletions
@@ -68,3 +68,14 @@ doc(
    permalink = "/docs/user_guide/platforms/",
    weight = "30",
 )
+
+doc(
+    name = "production",
+    src = "production.md",
+    category = "User Guide",
+    data = [
+        "sandboxing-tradeoffs.png",
+    ],
+    permalink = "/docs/user_guide/production/",
+    weight = "20",
+)
@@ -34,6 +34,10 @@ Yes. Please see the [Docker Quick Start][docker].

 Yes. Please see the [Kubernetes Quick Start][k8s].

+### How do I integrate gVisor in my existing production stack? {#productionize}
+
+See the [Production guide].
+
 ### What's the security model? {#security-model}

 See the [Security Model][security-model].
@@ -149,4 +153,5 @@ update containerd to 1.3.9 or 1.4.3 (or newer versions respectively).
 [filesystem]: /docs/user_guide/filesystem/
 [docker]: /docs/user_guide/quick_start/docker/
 [k8s]: /docs/user_guide/quick_start/kubernetes/
+[Production guide]: /docs/user_guide/production/
 [CVE-2020-15257]: https://github.com/containerd/containerd/security/advisories/GHSA-36xw-fx78-c5r4
@@ -1,10 +1,13 @@
 # Containerd Quick Start

 This document describes how to use `containerd-shim-runsc-v1` with the
-containerd runtime handler support on `containerd`.
+containerd runtime handler support on `containerd`. This is a similar setup as
+[GKE Sandbox], other than the
+[platform configuration](/docs/architecture_guide/platforms/).

-> ⚠️ NOTE: If you are using Kubernetes and set up your cluster using kubeadm you
-> may run into issues. See the [FAQ](../FAQ.md#runtime-handler) for details.
+> ⚠️ **Note**: If you are using Kubernetes and set up your cluster using
+> `kubeadm` you may run into issues. See the [FAQ](../FAQ.md#runtime-handler)
+> for details.

 ## Requirements

@@ -180,3 +183,13 @@ Verify that the Pod is running:
 ```shell
 kubectl get pod nginx-gvisor -o wide
 ```
+
+## What's next
+
+This setup is already done for you on [GKE Sandbox]. It is an easy way to get
+started with gVisor.
+
+Before taking this deployment to production, review the
+[Production guide](/docs/user_guide/production/).
+
+[GKE Sandbox]: https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods
@@ -4,8 +4,10 @@

 gVisor accesses the filesystem through a file proxy, called the Gofer. The gofer
 runs as a separate process, that is isolated from the sandbox. Gofer instances
-communicate with their respective sentry using the 9P protocol. For another
-explanation see [What is gVisor?](../README.md).
+communicate with their respective sentry using the 9P protocol.
+
+Configuring the filesystem provides performance benefits, but isn't the only
+step to optimizing gVisor performance. See the [Production guide] for more.

 ## Sandbox overlay

@@ -13,7 +15,8 @@ To isolate the host filesystem from the sandbox, you can set a writable tmpfs
 overlay on top of the entire filesystem. All modifications are made to the
 overlay, keeping the host filesystem unmodified.

-> Note: All created and modified files are stored in memory inside the sandbox.
+> **Note**: All created and modified files are stored in memory inside the
+> sandbox.

 To use the tmpfs overlay, add the following `runtimeArgs` to your Docker
 configuration (`/etc/docker/daemon.json`) and restart the Docker daemon:
@@ -58,3 +61,5 @@ Docker configuration (`/etc/docker/daemon.json`) and restart the Docker daemon:
    }
 }
 ```
+
+[Production guide]: ../production/
@@ -26,19 +26,21 @@ To download and install the latest release manually follow these steps:

 To install gVisor as a Docker runtime, run the following commands:

-```bash
-/usr/local/bin/runsc install
-sudo systemctl reload docker
-docker run --rm --runtime=runsc hello-world
+```shell
+$ /usr/local/bin/runsc install
+$ sudo systemctl reload docker
+$ docker run --rm --runtime=runsc hello-world
 ```

 For more details about using gVisor with Docker, see
-[Docker Quick Start](./quick_start/docker.md)
+[Docker Quick Start](./quick_start/docker.md). Please read the
+[Production guide](/docs/user_guide/production/) before running such a setup for
+production purposes.

-Note: It is important to copy `runsc` to a location that is readable and
-executable to all users, since `runsc` executes itself as user `nobody` to avoid
-unnecessary privileges. The `/usr/local/bin` directory is a good place to put
-the `runsc` binary.
+> **Note**: It is important to copy `runsc` to a location that is readable and
+> executable to all users, since `runsc` executes itself as user `nobody` to
+> avoid unnecessary privileges. The `/usr/local/bin` directory is a good place
+> to put the `runsc` binary.

 ## Install from an `apt` repository

@@ -8,6 +8,9 @@ control messages, and packet assembly — keeping it isolated from the host
 network stack. Data link layer packets are written directly to the virtual
 device inside the network namespace setup by Docker or Kubernetes.

+Configuring the network stack may provide performance benefits, but isn't the
+only step to optimizing gVisor performance. See the [Production guide] for more.
+
 The IP address and routes configured for the device are transferred inside the
 sandbox. The loopback device runs exclusively inside the sandbox and does not
 use the host. You can inspect them by running:
@@ -81,4 +84,3 @@ Offload (GSO) to run with a kernel that is newer than 3.17. Add the
    }
 }
 ```
-
@@ -5,6 +5,10 @@
 This guide described how to change the
 [platform](../architecture_guide/platforms.md) used by `runsc`.

+Configuring the platform provides significant performance benefits, but isn't
+the only step to optimizing gVisor performance. See the [Production guide] for
+more.
+
 ## Prerequisites

 If you intend to run the KVM platform, you will also to have KVM installed on
@@ -64,8 +68,8 @@ pass the `--platform` argument:
 You must restart the Docker daemon after making changes to this file, typically
 this is done via `systemd`:

-```bash
-sudo systemctl restart docker
+```shell
+$ sudo systemctl restart docker
 ```

 Note that you may configure multiple runtimes using different platforms. For
@@ -91,6 +95,7 @@ for the KVM platform:
 }
 ```

+[Production guide]: ../production/
 [nested-azure]: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/nested-virtualization
 [nested-gcp]: https://cloud.google.com/compute/docs/instances/enable-nested-virtualization-vm-instances
 [nested-virtualbox]: https://www.virtualbox.org/manual/UserManual.html#nested-virt
@@ -0,0 +1,150 @@
+# Production guide
+
+gVisor adds additional layers of defense to your containers, but comes with some
+performance overhead. This page discusses **best practices** how and where to
+integrate sandboxing within your production stack, to take full advantage of
+gVisor's security benefits while minimizing overhead.
+
+[TOC]
+
+## The role of sandboxing in your production stack {#role}
+
+At its core, gVisor sandboxes your containers, isolating them from the host's
+Linux kernel and from each other. This is relevant to your production stack for
+the following use-cases:
+
+*   Hardening **externally-reachable endpoints**, such as user-facing load
+    balancers, web servers, public API endpoints, etc.
+*   Providing **defense-in-depth protection** for critical workloads handling
+    sensitive information and/or under security compliance requirements, e.g.
+    payment processing, sensitive data analysis pipelines.
+*   Safely **operating a multi-tenancy environment** with security isolation,
+    such as when operating an app platform for multiple third-party customers.
+*   Providing additional features to your container stack, such as **intrusion
+    detection** and **checkpoint save/restore**.
+*   Safely **running untrusted code**, such as when running
+    third-party/user-provided code, or for software forensics. **Note**: This
+    guide is not appropriate for this use-case, and will instead focus on how to
+    run an existing **trusted** stack with gVisor.
+
+While gVisor is able to sandbox **any** application, it should generally not be
+used to sandbox **every** application.
+
+## Attack surface reduction {#attack-surface}
+
+Because sandboxing comes woth some performance overhead, you should first
+investigate ways to **reduce your outside attack surface** (without such
+overhead) as much as possible, prior to introducing sandboxing into your
+production stack at all.
+
+Consider running non-user-facing workloads in a separate virtual network, and
+only run the user-facing entry points into this network in a sandbox. You can
+also rely on network security with a service mesh like [Istio] to prevent
+network traffic from arriving at sensitive endpoints, though note that such
+solutions have [their own performance overhead][Istio overhead].
+
+If using a Cloud provider, consider **using your provider's hosted application
+solutions**, rather than rolling your own solution (sandboxed or not). This
+simultaneously reduces your ops burden, the overall attack surface you are
+personally responsible for, and most likely your overall Cloud provider charges.
+For example, if using Google Cloud, consider:
+
+*   [Cloud Spanner](https://cloud.google.com/spanner) as a database
+*   [Cloud Load Balancing](https://cloud.google.com/load-balancing) as load
+    balancer
+*   [Cloud Storage](https://cloud.google.com/storage) for static file serving
+
+These parts of your stack coincide with where sandboxing performance overhead is
+most prevalent, so keeping them outside of your sandboxed perimeter provides
+significant benefits.
+
+## Security/performance trade-off {#security-vs-performance}
+
+Once you've reduced your attack surface and have identified the components of
+your production stack that may benefit from sandboxing, you still need to
+determine whether the security benefits sandboxing provides are worth the
+performance overhead.
+
+gVisor protects your workload by intercepting system calls and emulating them in
+userspace. This shields the host Linux kernel and the sandboxed application from
+each other, **protecting against most Linux CVEs**, **container escape
+vulnerabilities**, and making **remote privilege-escalation attacks** less
+impactful. See [Security Model] for more details.
+
+On the other hand, sandboxing has a **performance penalty**. This overhead is
+multi-faceted and highly depends on the behavior of the workload being
+sandboxed. As a general guideline, **I/O-heavy** (*e.g. databases*) and
+**network-heavy** (*e.g. load balancers*) workloads will see degraded
+performance, whereas **CPU-bound** workloads (*e.g. API servers, non-static web
+servers, data pipelines*) will see minimal or no overhead. See
+[Performance Guide] for more details and data.
+
+Ultimately, the decision of whether to run each workload in a sandbox or not
+comes down to a **balancing decision** between:
+
+*   Sensitivity of your critical data
+*   Applicable compliance obligations and regulations
+*   Your organization's budget and PR risk tolerance
+*   Your application's performance requirements
+*   Overall security diligence
+
+The following diagram summarizes the security/performance tradeoff for various
+approaches of adding sandboxing to a typical stack.
+
+![Sandboxing tradeoff](sandboxing-tradeoffs.png "Sandboxing security/performance tradeoffs.")
+
+## Configuring gVisor for optimal performance {#configure-for-performance}
+
+Once you've identified the workloads that absolutely need sandboxing, it is
+worth spending some time to configure gVisor for optimal performance.
+
+### Choosing a platform {#configure-platform}
+
+gVisor supports multiple low-level implementations called Platforms (See
+[Platform architecture] for a detailed overview). Picking the right platform for
+your environment is the **highest-impact performance decision**.
+
+[GKE Sandbox] uses an optimized, custom platform which will provide good
+performance with no tuning required.
+
+When using gVisor outside of GKE Sandbox, we recommend **running gVisor on
+bare-metal machines** (not VMs). In such a setup, use the KVM platform for best
+performance.
+
+If you absolutely must run gVisor in a virtual machine, we recommend using the
+`ptrace` platform. This platform has the most flexibility, but its performance
+will lag behind that of KVM and the GKE Sandbox platform.
+
+<a class="button" href="/docs/user_guide/platforms/">Configure Platform
+&raquo;</a>
+
+### Optimizing I/O performance {#configure-io}
+
+**File I/O is typically the most impacted performance characteristic** of a
+gVisor-sandboxed workload. Because gVisor is a general-purpose sandbox, its
+default configuration must support all possible I/O interaction patterns.
+However, you can configure gVisor to use more aggressive caching policies where
+it makes sense.
+
+<a class="button" href="/docs/user_guide/filesystem/">Configure Filesystem
+&raquo;</a>
+
+### Optimizing network performance {#configure-network}
+
+**Networking is typically the second most-impacted performance characteristic**
+of a gVisor-sandboxed workload. gVisor implements its own network stack, which
+is optimized for security over performance. If your application is semi-trusted
+and network performance is paramount, you can optionally enable Network
+Passthrough to use the host's (Linux's) network stack, rather than gVisor's own.
+
+<a class="button" href="/docs/user_guide/networking/">Configure Networking
+&raquo;</a>
+
+[Istio]: https://istio.io/
+[Istio overhead]: https://istio.io/latest/docs/ops/deployment/performance-and-scalability/
+[Security Model]: /docs/architecture_guide/security/
+[Performance Guide]: /docs/architecture_guide/performance/
+[Platform architecture]: /docs/architecture_guide/platforms/
+[Spectre]: https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)
+[Denial-of-Service attacks]: https://httpd.apache.org/docs/trunk/misc/security_tips.html
+[GKE Sandbox]: https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods
@@ -21,7 +21,8 @@ MySQL via the `db` service host name.
 > [the I/O overhead](../../architecture_guide/performance) imposed by gVisor,
 > **it is not recommended to run your database in a sandbox**. The frontend is
 > the critical component with the largest outside attack surface, where gVisor's
-> security/performance trade-off makes the most sense.
+> security/performance trade-off makes the most sense. See the
+> [Production guide] for more details.

 > **Note**: Docker Compose uses it's own network by default and allows services
 > to communicate using their service name. Docker Compose does this by setting
@@ -101,9 +102,13 @@ Compose.

 Learn how to deploy [WordPress with Kubernetes][wordpress-k8s].

+Before deploying this to production, see the [Production guide] for how to take
+full advantage of gVisor.
+
 [docker-compose]: https://docs.docker.com/compose/
 [docker-install]: ../quick_start/docker.md
 [wordpress]: https://wordpress.com/
 [resolv.conf]: https://man7.org/linux/man-pages/man5/resolv.conf.5.html
 [wordpress-k8s]: kubernetes.md
 [compose-networking]: https://docs.docker.com/compose/networking/
+[Production guide]: /docs/user_guide/production/
@@ -18,7 +18,8 @@ containers: web server in the frontend, MySQL database in the backend.
 > [the I/O overhead](../../architecture_guide/performance) imposed by gVisor,
 > **it is not recommended to run your database in a sandbox**. The frontend is
 > the critical component with the largest outside attack surface, where gVisor's
-> security/performance trade-off makes the most sense.
+> security/performance trade-off makes the most sense. See the
+> [Production guide] for more details.

 First, let's define a few environment variables that are shared between both
 containers:
@@ -72,8 +73,12 @@ gVisor.
 Learn how to deploy WordPress with [Kubernetes][wordpress-k8s] or
 [Docker Compose][wordpress-compose].

+Before deploying this to production, see the [Production guide] for how to take
+full advantage of gVisor.
+
 [docker]: https://www.docker.com/
 [docker-install]: ../quick_start/docker.md
 [wordpress]: https://wordpress.com/
 [wordpress-k8s]: kubernetes.md
 [wordpress-compose]: docker-compose.md
+[Production guide]: /docs/user_guide/production/
@@ -32,8 +32,8 @@ gVisor** option. Select other options as you like:

 ### Check that gVisor is enabled

-The gvisor RuntimeClass is instantiated during node creation. You can check for
-the existence of the gvisor RuntimeClass using the following command:
+The gvisor `RuntimeClass` is instantiated during node creation. You can check
+for the existence of the gvisor `RuntimeClass` using the following command:

 ```shell
 $ kubectl get runtimeclass/gvisor
@@ -45,15 +45,16 @@ gvisor   gvisor    1h

 Now, let's deploy a WordPress site using GKE Sandbox. WordPress site requires
 two pods: web server in the frontend, MySQL database in the backend. Both
-applications use PersistentVolumes to store the site data data. In addition,
-they use secret store to share MySQL password between them.
+applications use `PersistentVolumes` to store the site data. In addition, they
+use secret store to share MySQL password between them.

 > **Note**: This example uses gVisor to sandbox the frontend web server, but not
 > the MySQL database backend. In a production setup, due to
 > [the I/O overhead](../../architecture_guide/performance) imposed by gVisor,
 > **it is not recommended to run your database in a sandbox**. The frontend is
 > the critical component with the largest outside attack surface, where gVisor's
-> security/performance trade-off makes the most sense.
+> security/performance trade-off makes the most sense. See the
+> [Production guide] for more details.

 First, let's download the deployment configuration files to add the runtime
 class annotation to them:
@@ -217,9 +218,9 @@ You are now ready to deploy the entire application. Just create a secret to
 store MySQL's password and *apply* both deployments:

 ```shell
-kubectl create secret generic mysql-pass --from-literal=password=${YOUR_SECRET_PASSWORD_HERE?}
-kubectl apply -f mysql-deployment.yaml
-kubectl apply -f wordpress-deployment.yaml
+$ kubectl create secret generic mysql-pass --from-literal=password=${YOUR_SECRET_PASSWORD_HERE?}
+$ kubectl apply -f mysql-deployment.yaml
+$ kubectl apply -f wordpress-deployment.yaml
 ```

 Wait for the deployments to be ready and an external IP to be assigned to the
@@ -241,7 +242,10 @@ Congratulations! You have just deployed a WordPress site using GKE Sandbox.
 To learn more about GKE Sandbox and how to run your deployment securely, take a
 look at the [documentation][gke-sandbox-docs].

+Before taking this deployment to production, review the [Production guide].
+
 [gke-sandbox-docs]: https://cloud.google.com/kubernetes-engine/docs/how-to/sandbox-pods
 [gke-sandbox]: https://cloud.google.com/kubernetes-engine/sandbox/
 [project-selector]: https://console.cloud.google.com/projectselector/kubernetes
 [wordpress]: https://wordpress.com/
+[Production guide]: /docs/user_guide/production/
@@ -156,6 +156,7 @@ docs(
        "//g3doc/user_guide:install",
        "//g3doc/user_guide:networking",
        "//g3doc/user_guide:platforms",
+        "//g3doc/user_guide:production",
        "//g3doc/user_guide/containerd:configuration",
        "//g3doc/user_guide/containerd:containerd_11",
        "//g3doc/user_guide/containerd:quick_start",