The Jul 21-22 incident: a commit thread wedged inside the change-notification
fan-out while holding _changeListeners_l/_dbs_l, every other commit thread piled
up behind it, and the PubSub listeners kept acking member/network changes into a
queue nothing drained -- silently losing them (CV1 authorize never applied).
- Lock hygiene: snapshot-then-call in DB::_memberChanged/_networkChanged,
DBMirrorSet fan-out, and ENC onNetworkUpdate/onNetworkMemberUpdate/
onNetworkMemberDeauthorize; no listener callback runs under notification locks
- Liveness watchdog in heartbeat(): warn when a commit thread makes no progress
for 60s, abort the process at 5min so the orchestrator restarts a zombie
- Ack gating: PubSub listeners nack (redeliver) instead of acking when the
commit pipeline is stalled or the queue is over depth
- libpq connect_timeout/keepalives/tcp_user_timeout/statement_timeout on pooled
connections so a dead DB peer throws instead of blocking a thread forever
- Name controller threads (wchan/stack dumps become readable); commit queue
depth and enqueue-to-commit latency gauges
_enabled is set to true by setUpPostDecodeReceiveThreads(), so disabled
until then, but the constructor wasn't initializing it. _concurrency is
not being used before being set but for safety's sake, ensure it has a
starting value as well.
Also, remove the vestigial _rxThreadCount, which is no longer used.
Muuuuch easier to use external dependencies now
Also tried out conan and vcpkg. Ran into dependency issues when solving for packages to install with conan.
vcpkg is just obtuse as all hell to install and not easy to integrate
Grant Limberg
dimitriy-dade
Grant Limberg
Chris Spiegel
Adam Ierymenko
Joseph Henry