gVisor currently implements CPU clocks as follows:
- A per-sentry "CPU clock ticker goroutine"
(task_sched.go:Kernel.runCPUClockTicker()) periodically advances
Kernel.cpuClock, causing it to serve as a very coarse but inexpensive
monotonic wall clock (that happens to be suspended when no tasks are
running).
- Task goroutines observe the most recent value of Kernel.cpuClock when
changing state (Task.gosched.Timestamp), and use it to compute the number of
CPU clock ticks that have elapsed in a given state. Thus, task CPU clocks are
approximately based on the wall time during which they were marked as
running.
- ITIMER_VIRTUAL, ITIMER_PROF, and RLIMIT_CPU are checked by the CPU clock
ticker goroutine after advancing Kernel.cpuClock. POSIX interval timers and
timerfds check CPU clocks (taskClock/tgClock) in ktime.SampledTimer
goroutines.
This has three major problems:
- ktime.SampledTimer goroutines for CPU clock timers run concurrently with the
CPU clock ticker, and are not informed as to when corresponding tasks start
or stop running (due to overhead on the task execution critical path), so
they can't determine when CPU clocks have/will advance; instead, they simply
poll CPU clocks on a period equal to that of the represented timer, resulting
in significant overhead for CPU-clock-based POSIX interval timers and
timerfds.
- For the same reason, CPU clock interval timers and timerfds may expire much
later than when the CPU clock is actually incremented; in the interval timer
case, this can result in notification signals being sent long after tasks
have stopped running. (This is the same problem as in b/116538398, which
motivated the special-casing of ITIMER_VIRTUAL and ITIMER_PROF described
above, but applied to POSIX interval timers.)
- The sentry does not impose a limit on the number of tasks that may be
concurrently marked running, so if more tasks are marked running than the
number of CPUs advertised to applications, application CPU utilization can
appear to exceed 100%.
This CL fixes these problems by introducing explicit per-Task and ThreadGroup
CPU clocks, directly advancing (up to Kernel.applicationCores of) them in the
CPU clock ticker, and directly expiring CPU timers when doing so. Itimer and
RLIMIT_CPU timers lose their special-casing and instead behave like other CPU
timers (see task_acct.go). Kernel.cpuClock is still required, but only for the
sentry watchdog.
Minor cleanup changes:
- Gather all stateify hooks in kernel_state.go.
- Replace kernel.randInt31n() with math/rand/v2, which fixes the same problem
(https://go.dev/blog/randv2#problem.rand).
Test workload:
```
#include <err.h>
#include <signal.h>
#include <time.h>
#include <chrono>
#include <thread>
constexpr int kNumTimers = 1000;
constexpr long kTimerPeriodNS = 10000000;
int main(int argc, char** argv) {
for (int i = 0; i < kNumTimers; i++) {
struct sigevent sev = {.sigev_notify = SIGEV_NONE};
timer_t timerid;
if (timer_create(CLOCK_THREAD_CPUTIME_ID, &sev, &timerid) < 0) {
err(1, "timer_create failed");
}
struct itimerspec it = {
.it_interval = {0, kTimerPeriodNS},
.it_value = {0, kTimerPeriodNS},
};
if (timer_settime(timerid, 0, &it, nullptr) < 0) {
err(1, "timer_settime failed");
}
}
std::this_thread::sleep_for(std::chrono::seconds(5));
return 0;
}
```
Before this CL:
```
# /usr/bin/time ./runsc --ignore-cgroups --platform kvm --network none do $(pwd)/workloads/threadcputimers
1.50user 0.17system 0:05.25elapsed 31%CPU (0avgtext+0avgdata 35792maxresident)k
0inputs+184outputs (10major+20889minor)pagefaults 0swaps
```
After this CL:
```
# /usr/bin/time ./runsc --ignore-cgroups --platform kvm --network none do $(pwd)/workloads/threadcputimers
0.10user 0.12system 0:05.22elapsed 4%CPU (0avgtext+0avgdata 34040maxresident)k
0inputs+192outputs (6major+20929minor)pagefaults 0swaps
```
PiperOrigin-RevId: 695198313
It is an idea of running codespell as part of our presubmit checks.
Before enabling it for new changes, let's fix what it has found.
Signed-off-by: Andrei Vagin <avagin@gmail.com>
This introduces a `metric.FieldValue` struct type that wraps a string.
All metric interfaces that deal with field values have been updated to use
pointers to this type instead of strings.
The intent of this change is to make it more obvious that field values must
be passed using references. Prior to this change, this was done using string
pointer comparisons. Now this must be done by using a pointer to the same
`metric.FieldValue` struct.
The struct type still externally exposes its string so that it can be referred
to in value function callbacks by "custom" metrics. (Though there are no
current uses of callback metrics with fields.)
PiperOrigin-RevId: 527030738
This optimizes both increments and lookup operations.
It does so using the following:
- For metrics of potential cardinality larger than 48, it will switch to using
a map rather than linear search for mapping field value combinations to the
index that combination corresponds to in the flattened list of values.
For metrics of cardinality 48 or smaller, linear search is still used as it
is still faster (and also still faster than binary search), as determined by
benchmarks.
- All field values are checked for pointer uniqueness (i.e. the pointer to the
start of each field value string must be unique). This is then used for
faster matching: instead of comparing whole strings (and needing to hash
them, in the case of doing map-based lookups), it compares pointer values.
In the context of this metric library, because all field values must be
pre-declared ahead of time, keeping references to these pre-declared strings
should always be possible. This is enforced: it will `panic` if the metric
user does not do this. In practice, this is easy to do by using `const`
strings for all metric values. This change does just that for existing
metrics with string fields.
From benchmarks, this speeds up the time to take a snapshot of existing
metrics by -8.35%. With the unimplemented syscall counter metric, this
optimization reduces the slowdown of adding this metric from +4,250% to
a still-large but much more manageable +290%. A further optimization
(cl/524419591) will reduce this overhead further before re-introducing the
unimplemented syscall counter metric.
PiperOrigin-RevId: 526134756
Use reflection and tags to provide automatic conversion from
Config to flags. This makes adding new flags less error-prone,
skips flags using default values (easier to read), and makes
tests correctly use default flag values for test Configs.
Updates #3494
PiperOrigin-RevId: 328662070
The meaning for skipDump was reversed, but not all callers
were updated. Change the meaning once again to forceDump, so
that the period between stack dump is respected from all
callers.
PiperOrigin-RevId: 308674373
This is needed for syzkaller to proper classify issues.
Right now, all watchdog issues are duped to one with the
subject "panic: Sentry detected stuck task(s). See stack
trace and message above for more details".
PiperOrigin-RevId: 297975363
* Rename syncutil to sync.
* Add aliases to sync types.
* Replace existing usage of standard library sync package.
This will make it easier to swap out synchronization primitives. For example,
this will allow us to use primitives from github.com/sasha-s/go-deadlock to
check for lock ordering violations.
Updates #1472
PiperOrigin-RevId: 289033387
The watchdog currently can find stuck tasks, but has no way to tell if the
sandbox is stuck before the application starts executing.
This CL adds a startup timeout and action to the watchdog. If Start() is not
called before the given timeout (if non-zero), then the watchdog will take the
action.
PiperOrigin-RevId: 277970577
Jamie LiuandgVisor bot
Andrei Vagin
Etienne PerotandgVisor bot
Adin ScannellandgVisor bot
Kevin KrakauerandgVisor bot
Ayush RanjanandgVisor bot
Nayana BidariandgVisor bot
Fabricio VoznikaandgVisor bot
Dean DengandgVisor bot
Nicolas LacasseandgVisor bot
Michael PrattandgVisor bot
Andrei VaginandgVisor bot
Ian GudgerandgVisor bot