This adds log lines (in the Sentry logs) and warnings (in the metricsviz HTML
page) in case the profiling process encountered significant measurement
errors.
PiperOrigin-RevId: 648569118
This is logged in the profiling metrics log and shown in the metrics
visualization page, with warnings and suggested fixes depending on the type
of measurement error that occurs.
The errors measured are:
- Missed snapshots, as detected by the number of expected snapshots given
the time the collection goroutine ran vs its expected profiling rate.
- Buffer overruns, as detected by the presence of backoff sleeps.
- Collection timing error, as detected by the amount of time elapsed
between the first and the last metric in a single collection cycle
as a fraction of the profiling rate.
- Sleep timing error, as detected by the amount of time between the
expected collection cycle start time and the actual collection cycle
start time, as a fraction of the profiling rate.
PiperOrigin-RevId: 648543674
Prior to this change, if `runsc` was configured to use a specific
`--profiling-metrics-log` path, then the second run of this runtime would
overwrite existing metric data in-place, which produces a corrupt file.
This change truncates the existing log, so that the resulting log actually
correspond to what just ran.
This only happens when the `os.File` supports `truncate(2)`, which stdout
does not, so the error is ignored if there is one. It's a best-effort thing.
PiperOrigin-RevId: 648493697
This turns the uint64 metric constructor arguments into a struct, making it
more explicit as to what each part means. It also allows the creation of
non-cumulative uint64 (gauge) metrics, and adds methods to decrement or set
them.
PiperOrigin-RevId: 647134245
This includes multiple improvements which, when combined, get me a clean
run on a long-running benchmark like `bazel_test` with a 100μs sample rate.
This includes:
- Not using `time.Sleep` for short durations (this made the most impact),
instead it either spins or yields.
- Adaptive initially-faster backoff time that increases exponentially if
it finds that it's not not sleeping long enough, rather than 100x
profiling rate backoff right off the bat.
- Larger buffers.
- More precise computation for duration by using a mix of `nanotime` and
actual `time.Now` readjusted once every N collections.
- Using an atomic boolean for checking if metric collection should be stopped
(should be cheaper than checking if a channel has data).
- Putting the ringbuffer item slice being processed on the collector/writer
stacks rather than dereferencing it for each cell.
PiperOrigin-RevId: 634123127
This allows detecting data corruption more finely.
A future change will make these errors skipped over, allowing data to
still be visualized even if partially corrupt.
PiperOrigin-RevId: 634038986
This is part of a series of changes to add metric charts in performance
benchmarks.
This is intended to increase the reliability of logs-based profiling metrics
by ensuring that Docker keeps enough of them around during long tests, or
tests where there are a lot of metrics being profiled, or tests with a very
fast profiling rate, or all of the above.
This also calls `sync` on the underlying logging file descriptor if possible.
Give a more helpful error messages if a log buffer overrun does occur.
PiperOrigin-RevId: 633389921
This is part of a series of changes to add metric charts in performance
benchmarks.
This helps make the logs self-contained by ensuring they carry the metric
metadata needed to chart them. Specifically, this helps the charting tool
know the meaning and type of metrics. The descriptions show up in the charts,
and the cumulative-ness determines whether the chart should show the data on
an absolute basis, or compute and show a delta-over-time.
PiperOrigin-RevId: 631322173
This is part of a series of changes to add metric charts in performance
benchmarks.
In contexts where profiling metrics are not being intermixed with other
logs, we do not have to prefix the data with a common prefix, nor do we
need the checksum. To keep profiling metric overhead to a minimum in these
cases, this distinguishes two types of writers:
- Normal buffered writers, which do nothing other than buffer writes to
the underlying writer.
- Line-buffered "lossy" writers, which do the prefixing and checksumming
that the code was previously doing.
Both can reuse the same underlying write loop (thanks Go generics!).
PiperOrigin-RevId: 631271157
This is part of a series of changes to add metric charts in performance
benchmarks.
Because profiling metric data may be injected within container logs, it can
find itself surrounded by output from the container logs which are not
profiling metric data. When using regular byte-size-based buffering, this
can lead to lines being merged together with unrelated output, resulting in
corrupted data.
The new writer implementation instead buffers write on a per-line-number
basis. The buffer is flushed when a certain number of lines has been reached.
The contents are also prefixed and suffixed by `\n` to ensure the data does
not share a line with the rest of the logs.
PiperOrigin-RevId: 631213716
This will allow us to check the integrity of the metrics data when read
out of a log file that may be polluted with other data, or truncated due
to overflowing a ringbuffer of logs somewhere in the logs pipeline.
This is part of a series of changes to add metric charts in performance
benchmarks.
PiperOrigin-RevId: 630487877
This is part of a series of changes to add metric charts in performance
benchmarks.
This change is useful to be able to extract profiling metric data easily
regardless of runtime configuration.
This change also changes where profiling metrics are initialized and
configured. They are now only part of `runsc boot`, rather than all `runsc`
invocations.
PiperOrigin-RevId: 629900287
Earlier the atomicbitops_state_autogen.go was not being built because it had
an impossible build condition: `(amd64 || arm64) && !amd64 && !arm64`.
This is a known deficiency in go_stateify tool. This was solved via having
two identical files like 32b_32bit.go and 32b_64bit.go. Just piggyback this.
This changes also enhances `atomicbitops.Bool` in the following ways:
- Added RacyLoad(), RacyStore() and CompareAndSwap() for Bool to bring it up to
speed with the other types.
- Delegated the actual atomic operations work to the underlying Uint32.
- Cleaned up code with b32(); similar to what sync/atomic.Bool does.
PiperOrigin-RevId: 597607982
When using conditionally compiled metrics there was only one go_tag to rule them
all. That's kind of a shame because when profiling something specific it's
nicer to know that only your metrics that you care about are compiled in and
metrics that you don't care about aren't.
This CL makes it easy to create new profiling go_tags, by abstracting the
profiling metric init functions under two builder structs: RealMetricBuilder
and FakeMetricBuilder; these builder structs are the only things that need
to change between the metrics-ON or OFF versions of conditional metric files
(aside from the go_tag controlling conditional compilation of course).
Ideally having to create these files manually wouldn't be necessary at all,
and we could use a go_template build rule instead, but that doesn't support
replacing go:build tags AFAIK.
PiperOrigin-RevId: 571153131
The idea behind conditionally compiled metrics originally was to use them in
hotpaths for profiling purposes. This CL makes that possible by outputting
declared metrics in TSV format, which can be used to track custom events at
runtime in relatively high resolution.
Usage:
1. Optionally enable compilation of runsc conditionally-compiled metrics
by passing in condmetric_profiling to the Go tags.
2. Add these flags to runsc:
- [Required] --profiling-metrics-log=/tmp/some.csv
- [Optional] --profiling-metrics=/task/syscalls,/task/faults
- If this flag is not specified it will monitor all
conditionally-compiled metrics by default.
- [Optional] --profiling-metrics-rate-us=10000
Some future improvements:
- Flag to output a metric-difference between timestamps instead of
constant accumulation.
- Output a gnuplot command along with the data.
- Current monitoring resolution is limited by what time.Sleep allows.
This can be overcome by spinning/yielding when lower monitoring
rates are requested.
PiperOrigin-RevId: 560849611
When debugging is enabled, metric updates are written to the log as they
are emitted. This change adds metric fields to this log, and formats values
better between integers-typed and distribution-typed metrics.
Sample output:
```
metric.go] Emitting metrics:
[...]
metric.go] Metric /netstack/udp/packets_sent: 0
metric.go] Metric /netstack/udp/receive_buffer_errors: 0
metric.go] Metric /netstack/udp/unknown_port_errors: 0
metric.go] Metric /trace/sessions_created: 0
metric.go] Metric unimplemented_syscalls[sysno=334]: 8
```
PiperOrigin-RevId: 539206126
This plumbs the distribution statistics out to the data consumed from the
gVisor sandbox process by the `runsc metric-server` process.
PiperOrigin-RevId: 537990236
