Timed in the worker around the BulkApply call; reported on the per-batch
completion line and the exception line. Gives a direct client-observed
latency signal to compare against Bigtable's server-side metrics.
Replace AsyncBulkApply with synchronous BulkApply calls fanned out to a
small worker pool (8 threads, 500-row batches). The async client path
makes progress only on the connection's shared background event threads,
which stall for tens of seconds at a time in production (observed
2026-07-21: 26k-row flushes taking 3.5-8 minutes against a near-idle
cluster with server-side MutateRows p99 of 50-80ms, completions arriving
in window-sized bursts). A sync call is driven entirely by its own
thread and has months of healthy history in this environment.
Batches are built and their results processed on the calling thread, so
_lastNodeInfo and the re-queue list stay lock-free; workers only perform
the RPC and record the outcome.
Also stop dropping failed mutations: re-queue them for the next cycle
(idempotent replays via deterministic cell timestamps, backlog capped by
requeuePendingStatus) unless the status is INVALID_ARGUMENT. This
covers transient per-mutation failures the client does not retry
internally, e.g. ACCESS_TOKEN_EXPIRED and UNAVAILABLE.
A stale .deps-build/otel CMake cache from an API-only run kept
WITH_API_ONLY=ON even under ZT_CONTROLLER_DEPS=1, silently installing an
API-only opentelemetry-cpp; the controller configure then fails with
"Component sdk is not a built component". Pass the flag explicitly so
the full build always overrides a cached value.
Bigtable's MutateRows API rejects requests with more than 100,000
mutations total, and google-cloud-cpp sends a BulkMutation as a single
request -- it never splits by mutation count. writePending() built one
batch from every pending check-in (up to 5 SetCells per row), so a cold
node_info cache after a controller restart could push a single flush far
past the limit (observed: 124582). INVALID_ARGUMENT is permanent, so the
whole batch failed and those check-ins were dropped for the cycle.
Flush in chunks of at most 25,000 mutations. Failure handling is now
per-chunk: FailedMutation::original_index() aligns against a chunk-local
row-hash vector, and if BulkApply throws, only the not-yet-written tail
is re-queued so already-applied chunks are not replayed.
Template was looking for {{networkId}} but that hadn't been populated
yet in the case where the IdP returned an error. This caused the
template parsing to crash and return its own error, disguising the error
from the IdP
Grant Limberg
Grant Limberg