This runs all benchmarks tagged as PGO-enabled, of which there is
currently just one for simplicity (the ffmpeg benchmark). All other
benchmarks are initially tagged out of PGO. I will send a different
change to enroll other benchmarks in PGO.
The make target runs each such benchmark and gathers profiles for each
benchmark. Multiple profile files for multiple runs of the same benchmark
are merged into one, then compared against the existing checked-in profile
used for PGO builds (which right now doesn't exist). If such a profile
doesn't exist or differs widely from the freshly-collected profile, then
this new profile is copied into the repository.
Such profiles are not used at all in builds yet, this is just the glue
that keeps them fresh in the repo.
PiperOrigin-RevId: 714311978
This is part of a series of changes to add metric charts in performance
benchmarks.
This change showcases how to process profiling metrics logs and is meant as
a demo. A future change will add it to other benchmarks.
PiperOrigin-RevId: 631694169
Fix the redis test after the image has been changed. The new image has a
different output format, so update the report function to accomodate.
This also means we can collect more metrics easily.
Also use the redis-cli to check that the server is serving instead of
wget --spider.
PiperOrigin-RevId: 558181384
The redis benchmark has inconsistent data due to timeouts. In general,
operations on a redis DB are pretty uniform WRT performance and having
a large set of operations is not useful for regression checks.
Add another test method with a smaller subset of operations and a filter in
the buildkite pipeline files.
PiperOrigin-RevId: 499574468
This CL uses container links rather than going over the host network stack.
This avoids relying on that network stack, making the benchmarks more reliable,
and simplifies benchmark setup code.
#codehealth
PiperOrigin-RevId: 486221868
Intermittenly, the connection between the client redis_benchmark container can be
flaky with the server, even if the server has been up for a long time. If this
happens, just re-run a client until we get a result.
Also, don't start a new server for each operation.
Also also, modify Makefile run-benchmark call to accept RUNTIME=runc correctly.
PiperOrigin-RevId: 378918886
- Tweak the benchmarks to work with b.N where appropriate. In many cases,
b.N was simply being ignored. This creates an implicit dependency in the
user passing a reasonable benchtime (less than or equal to the actual
runtime of the test, or using the X syntax) otherwise the test runs
forever.
- In cases where the above is impossible, explicitly set benchtime from
the test wrapper, to prevent the above behavior (tensorflow).
- Drop the *Reverse variants, which are simply hey benchmarks. We should
just add a hey benchmark. The platforms benchmarks already include a
native platform, and thus these benchmarks are incredibly confusing.
(In other words, BenchmarkNginxReverse has nothing to do with an nginx
benchmark for runsc.)
- Remove the redunant Harness object, which contains no state, in order
to slightly simplify the code.
- Make Block and Heap profiling actually work, but setting appropriate
runtime parameters (and plumbing them through the config).
- Split the profiling into two phases: start and stop, since some will
need to be started early, and others will need to happen at the end.
PiperOrigin-RevId: 349495377
Add parser and formatting for golang benchmarks for docker benchmarks.
Change adds a library for printing and parsing Test parameters and metrics.
Benchmarks use the library to print parameters in the Benchmark title
(e.g. the name field in b.Run()), and to report CustomMetrics. Parser
uses the library to parse printed data from benchmark output and
put it into BigQuery structs.
PiperOrigin-RevId: 336365628
This change:
- Ports the nginx benchmark.
- Switches the Httpd benchmark to use 'hey' as a client.
- Moves all parsers to their own package 'tools'.
Parsers are moved to their own package because 1) parsing output of a command
is often dependent on the format of the command (e.g. 'fio --json'), 2) to
enable easier reuse, and 3) clean up and simplify actual running benchmarks
(no TestParser functions and ugly sample output in benchmark files).
PiperOrigin-RevId: 324144165
Andrei Vagin
Etienne Perot
gVisor bot
Jing Chen
Zach Koopmans
Adin Scannell