mirror of
https://github.com/netbirdio/gvisor.git
synced 2026-05-22 17:12:49 -07:00
Proposal for runtime.DedicateOSThread().
Updates #2184 PiperOrigin-RevId: 346875966
This commit is contained in:
Jamie Liu
gVisor bot
@@ -0,0 +1,188 @@
|
||||
# `runtime.DedicateOSThread`
|
||||
|
||||
Status as of 2020-09-18: Deprioritized; initial studies in #2180 suggest that
|
||||
this may be difficult to support in the Go runtime due to issues with GC.
|
||||
|
||||
## Summary
|
||||
|
||||
Allow goroutines to bind to kernel threads in a way that allows their scheduling
|
||||
to be kernel-managed rather than runtime-managed.
|
||||
|
||||
## Objectives
|
||||
|
||||
* Reduce Go runtime overhead in the gVisor sentry (#2184).
|
||||
|
||||
* Minimize intrusiveness of changes to the Go runtime.
|
||||
|
||||
## Background
|
||||
|
||||
In Go, execution contexts are referred to as goroutines, which the runtime calls
|
||||
Gs. The Go runtime maintains a variably-sized pool of threads (called Ms by the
|
||||
runtime) on which Gs are executed, as well as a pool of "virtual processors"
|
||||
(called Ps by the runtime) of size equal to `runtime.GOMAXPROCS()`. Usually,
|
||||
each M requires a P in order to execute Gs, limiting the number of concurrently
|
||||
executing goroutines to `runtime.GOMAXPROCS()`.
|
||||
|
||||
The `runtime.LockOSThread` function temporarily locks the invoking goroutine to
|
||||
its current thread. It is primarily useful for interacting with OS or non-Go
|
||||
library facilities that are per-thread. It does not reduce interactions with the
|
||||
Go runtime scheduler: locked Ms relinquish their P when they become blocked, and
|
||||
only continue execution after another M "chooses" their locked G to run and
|
||||
donates their P to the locked M instead.
|
||||
|
||||
## Problems
|
||||
|
||||
### Context Switch Overhead
|
||||
|
||||
Most goroutines in the gVisor sentry are task goroutines, which back application
|
||||
threads. Task goroutines spend large amounts of time blocked on syscalls that
|
||||
execute untrusted application code. When invoking said syscall (which varies by
|
||||
gVisor platform), the task goroutine may interact with the Go runtime in one of
|
||||
three ways:
|
||||
|
||||
* It can invoke the syscall without informing the runtime. In this case, the
|
||||
task goroutine will continue to hold its P during the syscall, limiting the
|
||||
number of application threads that can run concurrently to
|
||||
`runtime.GOMAXPROCS()`. This is problematic because the Go runtime scheduler
|
||||
is known to scale poorly with `GOMAXPROCS`; see #1942 and
|
||||
https://github.com/golang/go/issues/28808. It also means that preemption of
|
||||
application threads must be driven by sentry or runtime code, which is
|
||||
strictly slower than kernel-driven preemption (since the sentry must invoke
|
||||
another syscall to preempt the application thread).
|
||||
|
||||
* It can call `runtime.entersyscallblock` before invoking the syscall, and
|
||||
`runtime.exitsyscall` after the syscall returns. In this case, the task
|
||||
goroutine will release its P while the syscall is executing. This allows the
|
||||
number of threads concurrently executing application code to exceed
|
||||
`GOMAXPROCS`. However, this incurs additional latency on syscall entry (to
|
||||
hand off the released P to another M, often requiring a `futex(FUTEX_WAKE)`
|
||||
syscall) and on syscall exit (to acquire a new P). It also drastically
|
||||
increases the number of threads that concurrently interact with the runtime
|
||||
scheduler, which is also problematic for performance (both in terms of CPU
|
||||
utilization and in terms of context switch latency); see #205.
|
||||
|
||||
- It can call `runtime.entersyscall` before invoking the syscall, and
|
||||
`runtime.exitsyscall` after the syscall returns. In this case, the task
|
||||
goroutine "lazily releases" its P, allowing the runtime's "sysmon" thread to
|
||||
steal it on behalf of another M after a 20us delay. This mitigates the
|
||||
context switch latency problem when there are few task goroutines and the
|
||||
interval between switches to application code (i.e. the interval between
|
||||
application syscalls, page faults, or signal delivery) is short. (Cynically,
|
||||
this means that it's most effective in microbenchmarks). However, the delay
|
||||
before a P is stolen can also be problematic for performance when there are
|
||||
both many task goroutines switching to application code (lazily releasing
|
||||
their Ps) *and* many task goroutines switching to sentry code (contending
|
||||
for Ps), which is likely in larger heterogeneous workloads.
|
||||
|
||||
### Blocking Overhead
|
||||
|
||||
Task goroutines block on behalf of application syscalls like `futex` and
|
||||
`epoll_wait` by receiving from a Go channel. (Future work may convert task
|
||||
goroutine blocking to use the `syncevent` package to avoid overhead associated
|
||||
with channels and `select`, but this does not change how blocking interacts with
|
||||
the Go runtime scheduler.)
|
||||
|
||||
If `runtime.LockOSThread()` is not in effect when a task goroutine blocks, then
|
||||
when the task goroutine is unblocked (by e.g. an application `FUTEX_WAKE`,
|
||||
signal delivery, or a timeout) by sending to the blocked channel,
|
||||
`runtime.ready` migrates the unblocked G to the unblocking P. In most cases,
|
||||
this implies that every application thread block/unblock cycle results in a
|
||||
migration of the thread between Ps, and therefore Ms, and therefore cores,
|
||||
resulting in reduced application performance due to loss of CPU caches.
|
||||
Furthermore, in most cases, the unblocking P cannot immediately switch to the
|
||||
unblocked G (instead resuming execution of its current application thread after
|
||||
completing the application's `futex(FUTEX_WAKE)`, `tgkill`, etc. syscall), often
|
||||
requiring that another P steal the unblocked G before it can resume execution.
|
||||
|
||||
If `runtime.LockOSThread()` is in effect when a task goroutine blocks, then the
|
||||
G will remain locked to its M, avoiding the core migration described above;
|
||||
however, wakeup latency is significantly increased since, as described in
|
||||
"Background", the G still needs to be selected by the scheduler before it can
|
||||
run, and the M that selects the G then needs to transfer its P to the locked M,
|
||||
incurring an additional `FUTEX_WAKE` syscall and round of kernel scheduling.
|
||||
|
||||
## Proposal
|
||||
|
||||
We propose to add a function, tentatively called `DedicateOSThread`, to the Go
|
||||
`runtime` package, documented as follows:
|
||||
|
||||
```go
|
||||
// DedicateOSThread wires the calling goroutine to its current operating system
|
||||
// thread, and exempts it from counting against GOMAXPROCS. The calling
|
||||
// goroutine will always execute in that thread, and no other goroutine will
|
||||
// execute in it, until the calling goroutine has made as many calls to
|
||||
// UndedicateOSThread as to DedicateOSThread. If the calling goroutine exits
|
||||
// without unlocking the thread, the thread will be terminated.
|
||||
//
|
||||
// DedicateOSThread should only be used by long-lived goroutines that usually
|
||||
// block due to blocking system calls, rather than interaction with other
|
||||
// goroutines.
|
||||
func DedicateOSThread()
|
||||
```
|
||||
|
||||
Mechanically, `DedicateOSThread` implies `LockOSThread` (i.e. it locks the
|
||||
invoking G to a M), but additionally locks the invoking M to a P. Ps locked by
|
||||
`DedicateOSThread` are not counted against `GOMAXPROCS`; that is, the actual
|
||||
number of Ps in the system (`len(runtime.allp)`) is `GOMAXPROCS` plus the number
|
||||
of bound Ps (plus some slack to avoid frequent changes to `runtime.allp`).
|
||||
Corollaries:
|
||||
|
||||
* If `runtime.ready` observes that a readied G is locked to a M locked to a P,
|
||||
it immediately wakes the locked M without migrating the G to the readying P
|
||||
or waiting for a future call to `runtime.schedule` to select the readied G
|
||||
in `runtime.findrunnable`.
|
||||
|
||||
* `runtime.stoplockedm` and `runtime.reentersyscall` skip the release of
|
||||
locked Ps; the latter also skips sysmon wakeup. `runtime.stoplockedm` and
|
||||
`runtime.exitsyscall` skip re-acquisition of Ps if one is locked.
|
||||
|
||||
* sysmon does not attempt to preempt Gs that are locked to Ps, avoiding
|
||||
fruitless overhead from `tgkill` syscalls and signal delivery.
|
||||
|
||||
* `runtime.findrunnable`'s work stealing skips locked Ps (suggesting that
|
||||
unlocked Ps be tracked in a separate array). `runtime.findrunnable` on
|
||||
locked Ps skip the global run queue, work stealing, and possibly netpoll.
|
||||
|
||||
* New goroutines created by goroutines with locked Ps are enqueued on the
|
||||
global run queue rather than the invoking P's local run queue.
|
||||
|
||||
While gVisor's use case does not strictly require that the association is
|
||||
reversible (with `runtime.UndedicateOSThread`), such a feature is required to
|
||||
allow reuse of locked Ms, which is likely to be critical for performance.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
* Make the runtime scale well with `GOMAXPROCS`. While we are also
|
||||
concurrently investigating this problem, this would not address the issues
|
||||
of increased preemption cost or blocking overhead.
|
||||
|
||||
* Make the runtime scale well with number of Ms. It is unclear if this is
|
||||
actually feasible, and would not address blocking overhead.
|
||||
|
||||
* Make P-locking part of `LockOSThread`'s behavior. This would likely
|
||||
introduce performance regressions in existing uses of `LockOSThread` that do
|
||||
not fit this usage pattern. In particular, since `DedicateOSThread`
|
||||
transitions the invoker's P from "counted against `GOMAXPROCS`" to "not
|
||||
counted against `GOMAXPROCS`", it may need to wake another M to run a new P
|
||||
(that is counted against `GOMAXPROCS`), and the converse applies to
|
||||
`UndedicateOSThread`.
|
||||
|
||||
* Rewrite the gVisor sentry in a language that does not force userspace
|
||||
scheduling. This is a last resort due to the amount of code involved.
|
||||
|
||||
## Related Issues
|
||||
|
||||
The proposed functionality is directly analogous to `spawn_blocking` in Rust
|
||||
async runtimes
|
||||
[`async_std`](https://docs.rs/async-std/1.8.0/async_std/task/fn.spawn_blocking.html)
|
||||
and [`tokio`](https://docs.rs/tokio/0.3.5/tokio/task/fn.spawn_blocking.html).
|
||||
|
||||
Outside of gVisor:
|
||||
|
||||
* https://github.com/golang/go/issues/21827#issuecomment-595152452 describes a
|
||||
use case for this feature in go-delve, where the goroutine that would use
|
||||
this feature spends much of its time blocked in `ptrace` syscalls.
|
||||
|
||||
* This feature may improve performance in the use case described in
|
||||
https://github.com/golang/go/issues/18237, given the prominence of
|
||||
syscall.Syscall in the profile given in that bug report.
|
||||
Reference in New Issue
Block a user