netbird/gvisor
0
0
Fork 0
mirror of https://github.com/netbirdio/gvisor.git synced 2026-05-22 17:12:49 -07:00
Code Issues Packages Projects Releases Wiki Activity

Add documentation and tutorials for Runtime Monitoring

Browse Source 
Updates #4805

PiperOrigin-RevId: 467993824
This commit is contained in:
Fabricio Voznika
2022-08-16 12:29:59 -07:00
committed by gVisor bot
parent b195ca54f3
commit 919a20f179
3 changed files with 505 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/seccheck/README.md
+54
View File
@@ -0,0 +1,54 @@
This directory provides an example of a monitoring process that receives
connections from gVisor sandboxes and prints the traces to `stdout`. The example
contains two main files:
* server.cc: this is where `main()` and all the code is. It sets up a server
listening to a Unix-domain socket located at `/tmp/gvisor_events.sock` or a
configurable location via a command line argument.
* pod_init.json: this file contains the trace configuration that should be
passed to `runsc`. It can be done either via `--pod-init-config` flag or
using `runsc trace create` command. Note that the socket location is
specified in this file, in case you change it.
# Usage
Let's first start the server, which waits for new connections:
```shell
$ bazel run examples/seccheck:server_cc
Socket address /tmp/gvisor_events.sock
```
Here is a simple example using `runsc do`:
```shell
runsc --rootless --network=none --pod-init-config=examples/seccheck/pod_init.json do echo 123
```
Back at the server terminal, you can see the following traces being outputted:
```
Connection accepted
Start => id: "runsc-329739" cwd: "/home/fvoznika" args: "echo" args: "123"
E Open sysno: 257 fd: -100 pathname: "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/libc.so.6" flags: 524288
X Open exit { errorno: 2 } sysno: 257 fd: -100 pathname: "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/libc.so.6" flags: 524288
E Open sysno: 257 fd: -100 pathname: "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/libc.so.6" flags: 524288
X Open exit { errorno: 2 } sysno: 257 fd: -100 pathname: "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/libc.so.6" flags: 524288
...
TaskExit =>
Connection closed
```
Connection messages indicate when `runsc` connected and disconnected to/from the
server. Then there is a trace for container start and a few syscalls to
`open(2)` for searching libraries. You can change `pod_init.json` to configure
the trace session to your liking.
To set this up with Docker, you can add the `--pod-init-config` flag when the
runtime is installed:
```shell
$ sudo runsc install --runtime=runsc-trace -- --pod-init-config=$PWD/examples/seccheck/pod_init.json
$ sudo systemctl restart docker
$ docker run --rm --runtime=runsc-trace hello-world
```
pkg/sentry/seccheck/README.md
+365
View File
@@ -0,0 +1,365 @@
# Introduction
This package provides a remote interface to observe behavior of the application
running inside the sandbox. It was built with runtime monitoring in mind, e.g.
threat detection, but it can be used for other purposes as well. It allows a
process running outside the sandbox to receive a stream of trace data
asynchronously. This process can watch actions performed by the application,
generate alerts when something unexpected occurs, log these actions, etc.
First, let's go over a few concepts before we get into the details.
## Concepts
- **Points:** these are discrete places (or points) in the code where
instrumentation was added. Each point has a unique name and schema. They can
be individually enabled/disabled. For example, `container/start` is a point
that is fired when a new container starts.
- **Point fields:** each point may contain fields that carry point data. For
example, `container/start` has a `id` field with the container ID that is
getting started.
- **Optional fields:** each point may also have optional fields. By default
these fields are not collected and they can be manually set to be collected
when the point is configured. These fields are normally more expensive to
collect and/or large, e.g. resolve path to FD, or data for read/write.
- **Context fields:** these are fields generally available to most events, but
are disabled by default. Like optional fields, they can be set to be
collected when the point is configured. Context field data comes from
context where the point is being fired, for example PID, UID/GID, container
ID are fields available to most trace points.
- **Sink:** sinks are trace point consumers. Each sink is identified by a name
and may handle trace points differently. Later we'll describe in more
detailed what sinks are available in the system and how to use them.
- **Session:** trace session is a set of points that are enabled with their
corresponding configuration. A trace session also has a list of sinks that
will receive the trace points. A session is identified by a unique name.
Once a session is deleted, all points belonging to the session are disabled
and the sinks destroyed.
If you're interested in exploring further, there are more details in the
[design doc](https://docs.google.com/document/d/1RQQKzeFpO-zOoBHZLA-tr5Ed_bvAOLDqgGgKhqUff2A/edit).
# Points
Every trance point in the system is identified by a unique name. The naming
convention is to scope the point with a main component followed by its name to
avoid conflicts. Here are a few examples:
- `sentry/signal_delivered`
- `container/start`
- `syscall/openat/enter`
> Note: the syscall trace point contains an extra level to separate the
> enter/exit points.
Most of the trace points are in the `syscall` component. They come in 2 flavors:
raw, schematized. Raw syscalls include all syscalls in the system and contain
the 6 arguments for the given syscall. Schematized trace points exist for many
syscalls, but not all. They provide fields that are specific to the syscalls and
fetch more information than is available from the raw syscall arguments. For
example, here is the schema for the open syscall:
```proto
message Open {
gvisor.common.ContextData context_data = 1;
Exit exit = 2;
uint64 sysno = 3;
int64 fd = 4;
string fd_path = 5;
string pathname = 6;
uint32 flags = 7;
uint32 mode = 8;
}
```
As you can see, come fields are in both raw and schemitized points, like `fd`
which is also `arg1` in the raw syscall, but here it has a name and correct
type. It also has fields like `pathname` that are not available in the raw
syscall event. In addition, `fd_path` is an optional field that can take the
`fd` and translate it into a full path for convenience. In some cases, the same
schema can be shared by many syscalls. In this example, `message Open` is used
for `open(2)`, `openat(2)` and `creat(2)` syscalls. The `sysno` field can be
used to distinguish between them. The schema for all syscall trace points can be
found
[here](https://cs.opensource.google/gvisor/gvisor/+/master:pkg/sentry/seccheck/points/syscall.proto).
Other components that exist today are:
* **sentry:** trace points fired from within gVisor's kernel
([schema](https://cs.opensource.google/gvisor/gvisor/+/master:pkg/sentry/seccheck/points/sentry.proto)).
* **container:** container related events
([schema](https://cs.opensource.google/gvisor/gvisor/+/master:pkg/sentry/seccheck/points/container.proto)).
The following command lists all trace points available in the system:
```shell
$ runsc trace metadata
POINTS (973)
Name: container/start, optional fields: [env], context fields: [time|thread_id|task_start_time|group_id|thread_group_start_time|container_id|credentials|cwd|process_name]
Name: sentry/clone, optional fields: [], context fields: [time|thread_id|task_start_time|group_id|thread_group_start_time|container_id|credentials|cwd|process_name]
Name: syscall/accept/enter, optional fields: [fd_path], context fields: [time|thread_id|task_start_time|group_id|thread_group_start_time|container_id|credentials|cwd|process_name]
...
```
> Note: the output format for `trace metadata` may change without notice.
The list above also includes what optional and context fields are available for
each trace point. Optional fields schema is part of the trace point proto, like
`fd_path` we saw above. Context fields are set in `context_data` field of all
points and is defined in
[gvisor.common.ContextData](https://cs.opensource.google/gvisor/gvisor/+/master:pkg/sentry/seccheck/points/common.proto;bpv=1;bpt=1;l=77?gsn=ContextData&gs=kythe%3A%2F%2Fgithub.com%2Fgoogle%2Fgvisor%3Flang%3Dprotobuf%3Fpath%3Dpkg%2Fsentry%2Fseccheck%2Fpoints%2Fcommon.proto%234.2).
# Sinks
Sinks receive enabled trace points and do something useful with them. They are
identified by a unique name. The same `runsc trace metadata` command used above
also lists all sinks:
```shell
$ runsc trace metadata
...
SINKS (2)
Name: remote
Name: null
```
> Note: the output format for `trace metadata` may change without notice.
## Remote
The remote sink serializes the trace point into protobuf and sends it to a
separate process. For threat detection, external monitoring processes can
receive connections from remote sinks and be sent a stream of trace points that
are occurring in the system. This sink connects to a remote process via
Unix-domain socket and expect the remote process to be listening for new
connections. If you're interested about creating a monitoring process that
communicates with the remote sink, [this document](sinks/remote/README.md) has
more details.
The remote sink has many properties that can be configured when its created
(more on how to configure sinks below):
* endpoint (mandatory): Unix-domain socket address to connect.
* retries: number of attempts to write the trace point before dropping it in
case the remote process is not responding. Note that a high number of
retries can significantly delay application execution.
* backoff: initial backoff time after the first failed attempt. This value
doubles every failed attempts up to the max.
* backoff_max: max duration to wait between retries.
## Null
The null sink does nothing with the trace points and it's used for testing.
Syscall tests enable all trace points, with all optional and context fields to
ensure there is no crash with them enabled.
## Strace (not implemented)
The strace sink has not been implemented yet. It's meant to replace the strace
mechanism that exists in the Sentry to simplify the code and add more trace
points to it.
> Note: It requires more than one trace session to be supported.
# Sessions
Trace sessions scope a set of trace point with their corresponding configuration
and a set of sinks that receives the points. Sessions can be created at sandbox
initialization time or during runtime. Creating sessions at init time guarantees
that no trace points are missed, which is important for threat detection. It is
configured using the `--pod-init-config` flag (more on it below). To manage
sessions during runtime, `runsc trace create|delete|list` is used to manipulate
trace sessions. Here are few examples assuming there is a running container with
ID=cont123 using Docker:
```shell
$ sudo runsc --root /run/docker/runtime-runc/moby trace create --config session.json cont123
$ sudo runsc --root /run/docker/runtime-runc/moby trace list cont123
SESSIONS (1)
"Default"
Sink: "remote", dropped: 0
$ sudo runsc --root /var/run/docker/runtime-runc/moby trace delete --name Default cont123
$ sudo runsc --root /var/run/docker/runtime-runc/moby trace list cont123
SESSIONS (0)
```
> Note: There is a current limitation that only a single session can exist in
> the system and it must be called `Default`. This restriction can be lifted in
> the future when more than one session is needed.
## Config
The event session can be defined using JSON for the `runsc trace create`
command. The session definition has 3 mains parts:
1. `name`: name of the session being created. Only `Default` for now.
1. `points`: array of points being enabled in the session. Each point has:
1. `name`: name of trace point being enabled.
1. `optional_fields`: array of optional fields to include with the trace
point.
1. `context_fields`: array of context fields to include with the trace
point.
1. `sinks`: array of sinks that will process the trace points.
1. `name`: name of the sink.
1. `config`: sink specific configuration.
1. `ignore_setup_error`: ignores failure to configure the sink. In the
remote sink case, for example, it doesn't fail container startup if the
remote process cannot be reached.
The session configuration above can also be used with the `--pod-init-config`
flag under the `"trace_session"` JSON object. There is a full example
[here](https://cs.opensource.google/gvisor/gvisor/+/master:examples/seccheck/pod_init.json)
> Note: For convenience, the `--pod-init-config` file can also be used with
> `runsc trace create` command. The portions of the Pod init config file that
> are not related to the session configuration are ignored.
# Full Example
Here, we're going to explore a how to use runtime monitoring end to end. Under
the `examples` directory there is an implementation of the monitoring process
that accepts connections from remote sinks and prints out all the trace points
it receives.
First, let's start the monitoring process and leave it running:
```shell
$ bazel run examples/seccheck:server_cc
Socket address /tmp/gvisor_events.sock
```
The server is now listening on the socket at `/tmp/gvisor_events.sock` for new
gVisor sandboxes to connect. Now let's create a session configuration file with
some trace points enabled and the remote sink using the socket address from
above:
```shell
$ cat <<EOF >session.json
{
"trace_session": {
"name": "Default",
"points": [
{
"name": "sentry/clone"
},
{
"name": "syscall/fork/enter",
"context_fields": [
"group_id",
"process_name"
]
},
{
"name": "syscall/fork/exit",
"context_fields": [
"group_id",
"process_name"
]
},
{
"name": "syscall/execve/enter",
"context_fields": [
"group_id",
"process_name"
]
},
{
"name": "syscall/sysno/35/enter",
"context_fields": [
"group_id",
"process_name"
]
},
{
"name": "syscall/sysno/35/exit"
}
],
"sinks": [
{
"name": "remote",
"config": {
"endpoint": "/tmp/gvisor_events.sock"
}
}
]
}
}
EOF
```
Now, we're ready to start a container and watch it send traces to the monitoring
process. The container we're going to create simply loops every 5 seconds and
writes something to stdout. While the container is running, we're going to call
`runsc trace` command to create a trace session.
```shell
# Start the container and copy the container ID for future reference.
$ docker run --rm --runtime=runsc -d bash -c "while true; do echo looping; sleep 5; done"
dee0da1eafc6b15abffeed1abc6ca968c6d816252ae334435de6f3871fb05e61
$ CID=dee0da1eafc6b15abffeed1abc6ca968c6d816252ae334435de6f3871fb05e61
# Create new trace session in the container above.
$ sudo runsc --root /var/run/docker/runtime-runc/moby trace create --config session.json ${CID?}
Trace session "Default" created.
```
In the terminal that you are running the monitoring process, you'll start seeing
messages like this:
```
Connection accepted
E Fork context_data { thread_group_id: 1 process_name: "bash" } sysno: 57
CloneInfo => created_thread_id: 110 created_thread_group_id: 110 created_thread_start_time_ns: 1660249219204031676
X Fork context_data { thread_group_id: 1 process_name: "bash" } exit { result: 110 } sysno: 57
E Execve context_data { thread_group_id: 110 process_name: "bash" } sysno: 59 pathname: "/bin/sleep" argv: "sleep" argv: "5"
E Syscall context_data { thread_group_id: 110 process_name: "sleep" } sysno: 35 arg1: 139785970818200 arg2: 139785970818200
X Syscall context_data { thread_group_id: 110 process_name: "sleep" } exit { } sysno: 35 arg1: 139785970818200 arg2: 139785970818200
```
The first message in the log is a notification that a new sandbox connected to
the monitoring process. The `E` and `X` in front of the syscall traces denotes
whether the trace belongs to an `E`nter or e`X`it syscall trace. The first
syscall trace shows a call to `fork(2)` from a process with `group_thread_id`
(or PID) equal to 1 and the process name is `bash`. In other words, this is the
init process of the container, running `bash`, and calling fork to execute
`sleep 5`. The next trace is from `sentry/clone` and informs that the forked
process has PID=110. Then, `X Fork` indicates that `fork(2)` syscall returned to
the parent. The child continues and executes `execve(2)` to call `sleep` as can
be seen from the `pathname` and `argv` fields. Note that at this moment, the PID
is 110 (child) but the process name is still `bash` because it hasn't executed
`sleep` yet. After `execve(2)` is called the process name changes to `sleep` as
expected. Next, it shows the `nanosleep(2)` raw syscalls which has `sysno`=35
(it's referred as `syscall/sysno/35` in the configuration file. One for enter
with the exit trace happening 5 seconds later.
Let's list all trace session that are active in the sandbox:
```shell
$ sudo runsc --root /var/run/docker/runtime-runc/moby trace list ${CID?}
SESSIONS (1)
"Default"
Sink: "remote", dropped: 0
```
It shows the `Default` session created above, using the `remote` sink and no
trace points have been dropped. Once we're done, the trace session can be
deleted with the command below:
```shell
$ sudo runsc --root /var/run/docker/runtime-runc/moby trace delete --name
Default ${CID?} Trace session "Default" deleted.
```
In the monitoring process you should see a message `Connection closed` to inform
that the sandbox has disconnected.
If you want to set up `runsc` to connect to the monitoring process automatically
before the application starts running, you can set the `--pod-init-config` flag
to the configuration file created above. Here's an example:
```shell
$ sudo runsc --install --runtime=runsc-trace -- --pod-init-config=$PWD/session.json
```
pkg/sentry/seccheck/sinks/remote/README.md
+86
View File
@@ -0,0 +1,86 @@
# Introduction
The remote sink implements a protocol that allows a remote process to receive a
stream of trace points being triggered inside the sandbox. The remote sink uses
Unix-domain socket (UDS) to connect to the remote process. The remote process is
expected to have already created the UDS and be listening to new connections.
This allows for a single process to monitor all sandboxes in the machine and
simplifies lifecycle management. When a new sandbox starts, it creates a new
connection. And when a sandbox exits, the connection is terminated.
# Security Considerations
It’s important to note that in gVisor’s Threat Model, the Sentry is not trusted.
In order to ensure a secure posture, we assume the worst and consider that the
Sentry has been exploited. With that in mind, the monitoring process must
validate and never trust input received from the Sentry because it can be
controlled by a malicious user. All fields must have hard coded size limits.
Each sandbox uses a dedicated socket to prevent a malicious container from
corrupting or DoS’ing other sandboxes communication.
Simplicity in the protocol is paramount to keep the code easy to audit and
secure. For this reason we chose to use UDS type `SOCK_SEQPACKET` to delimitate
message boundaries. Also, each message contains a header and the payload uses
[Protocol Buffers](https://developers.google.com/protocol-buffers) which is safe
to deserialize using standard libraries.
# Protocol
Upon a new connection, there is a handshake message to ensure that both sides
can communicate with each other. The handshake contract is detailed
[here](https://cs.opensource.google/gvisor/gvisor/+/master:pkg/sentry/seccheck/points/common.proto;drc=e06df74a657e01008194f905f2795d43dd5a825e;bpv=1;bpt=1;l=63?gsn=Handshake&gs=kythe%3A%2F%2Fgithub.com%2Fgoogle%2Fgvisor%3Flang%3Dprotobuf%3Fpath%3Dpkg%2Fsentry%2Fseccheck%2Fpoints%2Fcommon.proto%234.0).
This is the only time that the monitoring process writes to the socket. From
this point on, it only reads a stream of trace points generated from the Sentry.
Each message contain a header that describes the message being sent and a few
more control fields, e.g. number of messages dropped. There is a full
description of the header
[here](https://cs.opensource.google/gvisor/gvisor/+/master:pkg/sentry/seccheck/sinks/remote/wire/wire.go).
The payload can be deserialized based on the message type indicated in the
header, Each message type corresponds to a protobuf type defined in one of
[these files](https://cs.opensource.google/gvisor/gvisor/+/master:pkg/sentry/seccheck/points/).
# Compatibility
It’s important that updates to gVisor do not break compatibility with trace
consumers. They may not understand new events, or new event fields, but should
continue to work with the old event schema.
* **New message/trace point:** new messages and trace points can be added
freely. The monitoring process will fail when it tries to deserialize an
unknown proto type. They should ignore this error.
* **New field to event:** as long as proto updating rules are followed, the
monitoring process will be able to deserialize the event, ignoring new
fields.
* **Changes to existing fields:** these are rare given that syscall arguments
don’t change. But if this is necessary, it should be handled as a deletion
of the old field and addition of the new one. It may break event consumers
that are relying on the old field being set, but at least the event can be
deserialized and other fields will be correct. If possible, populate both
fields until consumers have migrated over.
* **Message header change:** similar to proto, header changes can only be
additional. Existing fields cannot change offsets. Header size can be used
to determine what portions of the header are available.
* **Change in wire format:** it requires changing protocol version. This will
be detected and handled during the handshake. If one of the side decide that
it cannot talk to the other side, the communication will terminate.
# Examples
If you're looking to create a new monitoring process, you can use any of the
examples provided as a starting point. As a picture is worth a thousand words,
the same applies for code examples:
1. **Go:**
[pkg/sentry/seccheck/sinks/remote/server/server.go](https://cs.opensource.google/gvisor/gvisor/+/master:pkg/sentry/seccheck/sinks/remote/server/server.go)
1. **C++:**
[examples/seccheck/README.md](../../../../../examples/seccheck/README.md)
# Testing
Apart from using `runsc` directly to test that your code works, you can use a
tool that we created to save and replay trace sessions in full without the need
for complex setup. Just run `runsc` once to capture the trace files you need for
the test, then just replay from the file as often as needed. See
[tracereplay](../../../../../tools/tracereplay/README.md) for more details.