Timed in the worker around the BulkApply call; reported on the per-batch
completion line and the exception line. Gives a direct client-observed
latency signal to compare against Bigtable's server-side metrics.
Replace AsyncBulkApply with synchronous BulkApply calls fanned out to a
small worker pool (8 threads, 500-row batches). The async client path
makes progress only on the connection's shared background event threads,
which stall for tens of seconds at a time in production (observed
2026-07-21: 26k-row flushes taking 3.5-8 minutes against a near-idle
cluster with server-side MutateRows p99 of 50-80ms, completions arriving
in window-sized bursts). A sync call is driven entirely by its own
thread and has months of healthy history in this environment.
Batches are built and their results processed on the calling thread, so
_lastNodeInfo and the re-queue list stay lock-free; workers only perform
the RPC and record the outcome.
Also stop dropping failed mutations: re-queue them for the next cycle
(idempotent replays via deterministic cell timestamps, backlog capped by
requeuePendingStatus) unless the status is INVALID_ARGUMENT. This
covers transient per-mutation failures the client does not retry
internally, e.g. ACCESS_TOKEN_EXPIRED and UNAVAILABLE.
Bigtable's MutateRows API rejects requests with more than 100,000
mutations total, and google-cloud-cpp sends a BulkMutation as a single
request -- it never splits by mutation count. writePending() built one
batch from every pending check-in (up to 5 SetCells per row), so a cold
node_info cache after a controller restart could push a single flush far
past the limit (observed: 124582). INVALID_ARGUMENT is permanent, so the
whole batch failed and those check-ins were dropped for the cycle.
Flush in chunks of at most 25,000 mutations. Failure handling is now
per-chunk: FailedMutation::original_index() aligns against a chunk-local
row-hash vector, and if BulkApply throws, only the not-yet-written tail
is re-queued so already-applied chunks are not replayed.
writePending() did a synchronous ReadRow per queued node to avoid
rewriting unchanged os/arch/version. With thousands of check-ins per
cycle that's thousands of sequential round-trips every 10s, and the read
was nearly pointless since last_seen and the IP cell are written every
cycle anyway.
Replace it with an in-process cache of the last-written node_info per
row (keyed by a 64-bit hash of the row key). node_info is emitted only
when changed, new, or past its refresh window; check_in is still written
every cycle. Failed rows are dropped from the cache so they're rewritten
next cycle, and stale entries are evicted (24h TTL). Net result: one
BulkApply per cycle, no read RPCs.
Grant Limberg