When compiling with `-c fastbuild`, the `runsc metric-server` subcommand will
be stubbed out, which makes compilation faster.
The `metricclient` library (used in tests that execute the metric server) is
updated to call the embedded metric server binary directly, rather than
relying on it being embedded in `runsc`.
PiperOrigin-RevId: 580000119
This creates a separate `go_binary` target for the metric server, and changes
the `runsc metric-server` subcommand to call it instead of directly executing
metric server code. This helps avoid Go library dependencies used only by the
metric server into the `runsc` binary.
PiperOrigin-RevId: 559221695
Prometheus's `scrape_config.metrics_path` setting allows customizing the
URL to scrape from its default of `/metrics`.
`runsc metric-server` supports optional `GET` URL query parameters to filter
the list of returned metrics, such that the scraping client can express which
metrics it's interested in to the metric server.
However, Prometheus's `scrape_config.metrics_path` encodes any URL-special
character to percent-encoding form, i.e. `?` becomes `%3F`. This is
interpreted by Go's HTTP server implementation as part of the URL path
component rather than a delimiter for the query parameters.
This CL makes `runsc metric-server` handle this case by internally rewriting
the URL back to `/metrics?query_string_here`.
While somewhat hacky, this makes Prometheus integration easier, and the
behavior of serving metrics data for URLs starting with `/metrics%3F` doesn't
seem like it would be surprising.
PiperOrigin-RevId: 523834900
This adds an optional `GET` parameter `runsc-sandbox-metrics-filter` which can
be used to filter the set of sandbox metrics requested from each sandbox. In
cases where the metric client isn't interested in learning everything about
the sandboxes, this saves on RPC cost and latency.
This also has the benefit of reducing memory usage of the `metric-server`
further, as it does not need to hold onto data that it will not export.
As the sandbox process is untrusted, the metrics returned are double-checked
on the `metric-server` side to ensure that the filter was respected.
PiperOrigin-RevId: 521055953
Prior to this change, the `--metric-server-allow-unknown-root` flag was set
at the `runsc` top level, which made sense for the case where this needed to
be set in e.g. Docker runtime flags with auto-spawn. Since this is no longer
the supported mode, having this flag at the top level no longer makes sense.
This change moves this flag to be a subcommand flag of `runsc metric-server`
instead, and renames it to `--allow-unknown-root` (since it's obvious it's
about the metric server now).
This change is backwards-incompatible to anyone who was specifying this flag,
but as this feature is very new and this flag was never documented, hopefully
this isn't much of a disruption.
Same deal for `--metric-exporter-prefix`, though this one is used in the
`runsc export-metrics` subcommand as well, so duplicated it in both places.
PiperOrigin-RevId: 514580178
Prior to this CL, the metric server generated its own ID when it discovered
a new sandbox. This means existing sandboxes get new iteration IDs, which
breaks the continuity of counter metrics across process restarts.
This CL uses the container creation time in the sandbox state file, and uses
this (together with the sandbox ID) to generate a unique ID for each
instantiation of a sandbox with a given ID.
Added test to verify this behavior.
PiperOrigin-RevId: 505208685
This adds static configuration information to what's exported about each
sandbox. In turn, this allows correlating Sentry metrics against salient
parts of its configuration (e.g. platform, network mode, etc.).
This is exported as key-value labels. This is a bit awkward, but is the only
option we have because Prometheus doesn't support string-typed metrics.
This change is part of a series of changes to support Prometheus-style metrics
in `runsc`.
Sample output (from `container_test`, newlines added manually for readability):
```
# HELP testmetric_meta_sandbox_metadata Key-value pairs about per-sandbox
# metadata.
# TYPE testmetric_meta_sandbox_metadata gauge
testmetric_meta_sandbox_metadata{coretags="false",
iterationid="8729125096080520243",
namespace="foons",
network="none",
platform="ptrace",
pod="foopod",
sandbox="test-container-RR2PYJOS5376N6S7SLA4KL6NQJOJVEHG"} 1 1672959320454
```
PiperOrigin-RevId: 504403468
This library can be used to query and start a `runsc metric-server`.
It is used during sandbox startup when metrics are requested, and also in
tests in order to query metrics data.
This change has no tests, but coverage is provided in a later change that
provides an end-to-end container tests that the metric server works and
exports data faithfully.
This change is part of a series of changes to support Prometheus-style metrics
in `runsc`. Doing so requires making several seemingly-odd design decisions,
due to the following architectural constraints:
- Prometheus requires an HTTP server serving the `/metrics` endpoint.
- For performance reasons, the `runsc boot` process cannot run the `netpoller`
goroutine.
- Since we don't want to write our own HTTP server implementation, this
means the HTTP endpoint has to be served by a separate process that
remains running during the lifetime of the container.
- The `runsc boot` process is untrusted.
- This means we cannot trust metrics data that comes out of the Sentry.
Therefore, there needs to be an elaborate dance where we pre-register
metric metadata before starting any untrusted workload. Then, the server
relaying the metric data must verify the validity of metric values against
this metric metadata. This avoids leaking metrics, cardinality blow-ups,
and other such DoS vectors.
- This feature needs to be easy-to-use in a typical Docker setting.
- This means having the ability to just say
`--metrics-server=localhost:1337` in the `runsc` runtime entry in
`/etc/docker/daemon.json` and have that Just Work(TM), even when multiple
containers are running.
- Since only one process may listen on a port at a given time, this means
the metric server needs to be able to multiplex requests out to multiple
running sandboxes, and remain alive for the entire duration of either of
these sandboxes.
- For this reason, the metrics server runs *outside* of the usual
per-container cgroups.
- This also saves system resources by not running one server per sandbox.
- The metrics server must be exposed to the outside world, and cannot assume
that its clients are trustworthy.
- For this reason, a metrics server is bound to a runtime root directory,
and double-checks all that the sandboxes it is asked to follow actually
exist in this root directory.
PiperOrigin-RevId: 503529381
Etienne Perot
Adin Scannell