Reduce spans emitted per controller request without touching the head
sample rate. Removes 20 StartSpan sites (97 -> 77) that traced the same
logical op at multiple layers:
- DB.cpp: drop all db:: read-helper/leaf spans; the db_mirror_set::
wrapper survives as the single span per read op (also removes the
latent member_id-from-networkId attribute bug).
- CentralDB.cpp: drop inner write-op spans (save/+children, nodeIsOnline,
eraseNetwork, eraseMember); keep getSSOAuthInfo (errors) and all
background-thread spans (commitThread, onlineNotificationThread, init).
- EmbeddedNetworkController.cpp: drop per-rule parseRule span and the
no-op _startThreads span; keep request/_request roots.
Sampler, trace-context propagation, commit/Bigtable-write spans, and all
business logic are unchanged.
- Log network/member ids on every inbound PubSub config ('has old/new config'
lines were untraceable without them), on the frontend-gate drops (previously
fully silent), on the invalid-id save skips (the one silent path where a
delivered-and-acked change could vanish), and on the 100-attempt commit drop
- commitThread member branch: missing network is a permanent condition (nodes
join-looping a deleted network) -- skip with one log line instead of the
COUNT+GROUP BY+one_row() form that threw on zero rows and turned each item
into a 100-attempt retry storm; NULL frontend now reads as ''
- cleanMember also strips change_source: request-path saves inherited the
frontend's provenance from the cached record, mis-attributing the write and
silently skipping the controller->Central echo (stable CV1/controller
divergence); the authorized-path save now tags change_source after cleaning
- Deny and SSO-pending paths persist the member only on first sighting or a
just-learned identity: the old per-request full-snapshot save could race
with and overwrite a concurrent authorize arriving via PubSub
A member auto-created by the controller started at revision 0; its first save
bumped to 1 and the echo to Central carried 1. A manually created CV1 member
row also defaults to revision 1, and Central's UpsertMemberFromController only
applies echoes with a strictly greater revision -- 1 < 1 never passed, so the
echo carrying the learned identity was skipped forever and manually created
members stayed identity=""/revision=1 in CV1. Starting at 1 (matching
Central's DB default) makes the first echo carry 2.
From adversarial review of the previous commit:
- CtlUtil: guard <pthread.h> for Windows (client build compiles the controller
sources) and define _GNU_SOURCE on Linux for the two-arg pthread_setname_np
under the strict -std=c++17 make build
- Zero _commitThreadLastActive before the heartbeat thread and PubSub listeners
start; C++17 atomics are indeterminate until stored (C++20 masked this in
local checks) and a garbage stamp could spuriously abort at startup
- Watchdog aborts only when every started commit thread is stalled past the
threshold (the incident signature), or one thread exceeds 3x -- a single slow
item (e.g. the deauth fan-out of a huge network deletion) can no longer kill
a healthy controller
- BlockingQueue: lock size() (raced with push/pop; now read cross-thread by
the health gate, watchdog, and depth gauge) and notify gc in the timed get()
to match the untimed variant
- Drop statement_timeout from the hardened conninfo: it also capped the
startup bulk COPY loads, where a timeout exits the process -> crash loop on
a slow/degraded DB; dead-peer hangs are covered by keepalives and
tcp_user_timeout, and it could clobber an operator-supplied options=
- Rate-limit the listener nack logs (every redelivery attempt passes through
the health gate during a stall)
The Jul 21-22 incident: a commit thread wedged inside the change-notification
fan-out while holding _changeListeners_l/_dbs_l, every other commit thread piled
up behind it, and the PubSub listeners kept acking member/network changes into a
queue nothing drained -- silently losing them (CV1 authorize never applied).
- Lock hygiene: snapshot-then-call in DB::_memberChanged/_networkChanged,
DBMirrorSet fan-out, and ENC onNetworkUpdate/onNetworkMemberUpdate/
onNetworkMemberDeauthorize; no listener callback runs under notification locks
- Liveness watchdog in heartbeat(): warn when a commit thread makes no progress
for 60s, abort the process at 5min so the orchestrator restarts a zombie
- Ack gating: PubSub listeners nack (redeliver) instead of acking when the
commit pipeline is stalled or the queue is over depth
- libpq connect_timeout/keepalives/tcp_user_timeout/statement_timeout on pooled
connections so a dead DB peer throws instead of blocking a thread forever
- Name controller threads (wchan/stack dumps become readable); commit queue
depth and enqueue-to-commit latency gauges
Timed in the worker around the BulkApply call; reported on the per-batch
completion line and the exception line. Gives a direct client-observed
latency signal to compare against Bigtable's server-side metrics.
Replace AsyncBulkApply with synchronous BulkApply calls fanned out to a
small worker pool (8 threads, 500-row batches). The async client path
makes progress only on the connection's shared background event threads,
which stall for tens of seconds at a time in production (observed
2026-07-21: 26k-row flushes taking 3.5-8 minutes against a near-idle
cluster with server-side MutateRows p99 of 50-80ms, completions arriving
in window-sized bursts). A sync call is driven entirely by its own
thread and has months of healthy history in this environment.
Batches are built and their results processed on the calling thread, so
_lastNodeInfo and the re-queue list stay lock-free; workers only perform
the RPC and record the outcome.
Also stop dropping failed mutations: re-queue them for the next cycle
(idempotent replays via deterministic cell timestamps, backlog capped by
requeuePendingStatus) unless the status is INVALID_ARGUMENT. This
covers transient per-mutation failures the client does not retry
internally, e.g. ACCESS_TOKEN_EXPIRED and UNAVAILABLE.
A stale .deps-build/otel CMake cache from an API-only run kept
WITH_API_ONLY=ON even under ZT_CONTROLLER_DEPS=1, silently installing an
API-only opentelemetry-cpp; the controller configure then fails with
"Component sdk is not a built component". Pass the flag explicitly so
the full build always overrides a cached value.
Bigtable's MutateRows API rejects requests with more than 100,000
mutations total, and google-cloud-cpp sends a BulkMutation as a single
request -- it never splits by mutation count. writePending() built one
batch from every pending check-in (up to 5 SetCells per row), so a cold
node_info cache after a controller restart could push a single flush far
past the limit (observed: 124582). INVALID_ARGUMENT is permanent, so the
whole batch failed and those check-ins were dropped for the cycle.
Flush in chunks of at most 25,000 mutations. Failure handling is now
per-chunk: FailedMutation::original_index() aligns against a chunk-local
row-hash vector, and if BulkApply throws, only the not-yet-written tail
is re-queued so already-applied chunks are not replayed.
Template was looking for {{networkId}} but that hadn't been populated
yet in the case where the IdP returned an error. This caused the
template parsing to crash and return its own error, disguising the error
from the IdP
Grant Limberg
Grant Limberg