Reduce spans emitted per controller request without touching the head
sample rate. Removes 20 StartSpan sites (97 -> 77) that traced the same
logical op at multiple layers:
- DB.cpp: drop all db:: read-helper/leaf spans; the db_mirror_set::
wrapper survives as the single span per read op (also removes the
latent member_id-from-networkId attribute bug).
- CentralDB.cpp: drop inner write-op spans (save/+children, nodeIsOnline,
eraseNetwork, eraseMember); keep getSSOAuthInfo (errors) and all
background-thread spans (commitThread, onlineNotificationThread, init).
- EmbeddedNetworkController.cpp: drop per-rule parseRule span and the
no-op _startThreads span; keep request/_request roots.
Sampler, trace-context propagation, commit/Bigtable-write spans, and all
business logic are unchanged.
- Log network/member ids on every inbound PubSub config ('has old/new config'
lines were untraceable without them), on the frontend-gate drops (previously
fully silent), on the invalid-id save skips (the one silent path where a
delivered-and-acked change could vanish), and on the 100-attempt commit drop
- commitThread member branch: missing network is a permanent condition (nodes
join-looping a deleted network) -- skip with one log line instead of the
COUNT+GROUP BY+one_row() form that threw on zero rows and turned each item
into a 100-attempt retry storm; NULL frontend now reads as ''
- cleanMember also strips change_source: request-path saves inherited the
frontend's provenance from the cached record, mis-attributing the write and
silently skipping the controller->Central echo (stable CV1/controller
divergence); the authorized-path save now tags change_source after cleaning
- Deny and SSO-pending paths persist the member only on first sighting or a
just-learned identity: the old per-request full-snapshot save could race
with and overwrite a concurrent authorize arriving via PubSub
A member auto-created by the controller started at revision 0; its first save
bumped to 1 and the echo to Central carried 1. A manually created CV1 member
row also defaults to revision 1, and Central's UpsertMemberFromController only
applies echoes with a strictly greater revision -- 1 < 1 never passed, so the
echo carrying the learned identity was skipped forever and manually created
members stayed identity=""/revision=1 in CV1. Starting at 1 (matching
Central's DB default) makes the first echo carry 2.
From adversarial review of the previous commit:
- CtlUtil: guard <pthread.h> for Windows (client build compiles the controller
sources) and define _GNU_SOURCE on Linux for the two-arg pthread_setname_np
under the strict -std=c++17 make build
- Zero _commitThreadLastActive before the heartbeat thread and PubSub listeners
start; C++17 atomics are indeterminate until stored (C++20 masked this in
local checks) and a garbage stamp could spuriously abort at startup
- Watchdog aborts only when every started commit thread is stalled past the
threshold (the incident signature), or one thread exceeds 3x -- a single slow
item (e.g. the deauth fan-out of a huge network deletion) can no longer kill
a healthy controller
- BlockingQueue: lock size() (raced with push/pop; now read cross-thread by
the health gate, watchdog, and depth gauge) and notify gc in the timed get()
to match the untimed variant
- Drop statement_timeout from the hardened conninfo: it also capped the
startup bulk COPY loads, where a timeout exits the process -> crash loop on
a slow/degraded DB; dead-peer hangs are covered by keepalives and
tcp_user_timeout, and it could clobber an operator-supplied options=
- Rate-limit the listener nack logs (every redelivery attempt passes through
the health gate during a stall)
The Jul 21-22 incident: a commit thread wedged inside the change-notification
fan-out while holding _changeListeners_l/_dbs_l, every other commit thread piled
up behind it, and the PubSub listeners kept acking member/network changes into a
queue nothing drained -- silently losing them (CV1 authorize never applied).
- Lock hygiene: snapshot-then-call in DB::_memberChanged/_networkChanged,
DBMirrorSet fan-out, and ENC onNetworkUpdate/onNetworkMemberUpdate/
onNetworkMemberDeauthorize; no listener callback runs under notification locks
- Liveness watchdog in heartbeat(): warn when a commit thread makes no progress
for 60s, abort the process at 5min so the orchestrator restarts a zombie
- Ack gating: PubSub listeners nack (redeliver) instead of acking when the
commit pipeline is stalled or the queue is over depth
- libpq connect_timeout/keepalives/tcp_user_timeout/statement_timeout on pooled
connections so a dead DB peer throws instead of blocking a thread forever
- Name controller threads (wchan/stack dumps become readable); commit queue
depth and enqueue-to-commit latency gauges
Timed in the worker around the BulkApply call; reported on the per-batch
completion line and the exception line. Gives a direct client-observed
latency signal to compare against Bigtable's server-side metrics.
Replace AsyncBulkApply with synchronous BulkApply calls fanned out to a
small worker pool (8 threads, 500-row batches). The async client path
makes progress only on the connection's shared background event threads,
which stall for tens of seconds at a time in production (observed
2026-07-21: 26k-row flushes taking 3.5-8 minutes against a near-idle
cluster with server-side MutateRows p99 of 50-80ms, completions arriving
in window-sized bursts). A sync call is driven entirely by its own
thread and has months of healthy history in this environment.
Batches are built and their results processed on the calling thread, so
_lastNodeInfo and the re-queue list stay lock-free; workers only perform
the RPC and record the outcome.
Also stop dropping failed mutations: re-queue them for the next cycle
(idempotent replays via deterministic cell timestamps, backlog capped by
requeuePendingStatus) unless the status is INVALID_ARGUMENT. This
covers transient per-mutation failures the client does not retry
internally, e.g. ACCESS_TOKEN_EXPIRED and UNAVAILABLE.
A stale .deps-build/otel CMake cache from an API-only run kept
WITH_API_ONLY=ON even under ZT_CONTROLLER_DEPS=1, silently installing an
API-only opentelemetry-cpp; the controller configure then fails with
"Component sdk is not a built component". Pass the flag explicitly so
the full build always overrides a cached value.
Bigtable's MutateRows API rejects requests with more than 100,000
mutations total, and google-cloud-cpp sends a BulkMutation as a single
request -- it never splits by mutation count. writePending() built one
batch from every pending check-in (up to 5 SetCells per row), so a cold
node_info cache after a controller restart could push a single flush far
past the limit (observed: 124582). INVALID_ARGUMENT is permanent, so the
whole batch failed and those check-ins were dropped for the cycle.
Flush in chunks of at most 25,000 mutations. Failure handling is now
per-chunk: FailedMutation::original_index() aligns against a chunk-local
row-hash vector, and if BulkApply throws, only the not-yet-written tail
is re-queued so already-applied chunks are not replayed.
Template was looking for {{networkId}} but that hadn't been populated
yet in the case where the IdP returned an error. This caused the
template parsing to crash and return its own error, disguising the error
from the IdP
* Brewfile for homebrew dependencies on macos
* Pull & build google-cloud-cpp/otel via bootstrap script rather than
keeping it all in the ext/ subfolder.
* Update controller CI build with new steps
writePending() did a synchronous ReadRow per queued node to avoid
rewriting unchanged os/arch/version. With thousands of check-ins per
cycle that's thousands of sequential round-trips every 10s, and the read
was nearly pointless since last_seen and the IP cell are written every
cycle anyway.
Replace it with an in-process cache of the last-written node_info per
row (keyed by a 64-bit hash of the row key). node_info is emitted only
when changed, new, or past its refresh window; check_in is still written
every cycle. Failed rows are dropped from the cache so they're rewritten
next cycle, and stale entries are evicted (24h TTL). Net result: one
BulkApply per cycle, no read RPCs.
- Default enableBroadcast to true on load (initializeNetworks and _getNetwork) to match DB::initNetwork. Networks whose JSONB lacks the field now get broadcast on instead of off, which is the documented ZeroTier default.
- Stop overwriting networks_ctl.frontend on every UPDATE. It should remain whatever was set at INSERT. The early-exit guard already prevents cross-frontend writes from applying, so dropping the SET preserves correctness. Members are unaffected — they source frontend from the network row, not the change payload.
- Add enableBroadcast, creationTime, and revision to PubSubListener's network toJson. The protobuf carries them, but the listener was dropping them on inbound, which caused revision to reset on every PubSub-delivered network update.
- Rename member version keys to vMajor/vMinor/vRev/vProto in both PubSubListener (incoming) and PubSubWriter (outgoing). CentralDB, EmbeddedNetworkController, and the rest of the codebase already use these names; the listener/writer were the only sites using versionMajor/Minor/Rev/Protocol, so member version info was being silently dropped through PubSub in both directions.
- Pass newMember by const reference in ControllerChangeNotifier::notifyMemberChange to match notifyNetworkChange. Avoids a full JSON copy on every member notify.
- Remove duplicate nlohmann::json::parse call in RedisNetworkListener.
ssoConfig checked wrong variable
DNS missing-branch wrote to input not output
SSO fields renamed to sso* and nested under ssoConfig (across CentralDB, DB defaults, migration_models)
Member identity populated into the right field; address = memberId
noAutoAssignIPs → noAutoAssignIps casing
Member version columns now SELECTed + populated (both sites)
removeTraceLevel typo in DB.cpp defaults
8. eraseMember uninitialized tmp2 — member deletes now actually delete
9. Connection leak on continue in both commit branches
10. Unsafe (uint64_t)config["revision"] cast → OSUtils::jsonInt
11. target = "NULL" literal text → std::optional<std::string> for real SQL NULL
~13 unsafe direct json→string casts → OSUtils::jsonString; dead vars removed from save(); change_source blocks use is_string() checks.
Heartbeat zadd literal "controllerId" → variable
Network revision double-increment removed
PubSub notifier now sees actual old state (DB query moved before INSERT in both branches)
_delete_network reads config["id"] instead of config["nwid"]
ip_assignments empty-string guards added at both sites (also closes G)
ensure last_modified is set on every write to networks & members
The 10-second session.cancel() loop raced with in-flight acks — when
cancel fired while the GCP client was processing messages, acks were
lost before reaching the server. With message ordering enabled, an
unacked message blocks all subsequent messages on that ordering key,
causing silent stalls with no error output.
Two fixes:
- Replace the cancel/reconnect timer with a blocking session.get(),
storing the session future so the destructor can cancel on shutdown.
- Always ack messages even when onNotification fails — permanent errors
(bad protobuf, missing fields) will never succeed on retry and would
otherwise poison the ordering key indefinitely.
Allow controllers to advertise which central version (cv1, cv2, or all)
they are assigned to handle via a new configurable field. The value is
persisted to the database on each heartbeat and validated at startup
against the DB CHECK constraint.
Publish CTL_NONCE_UPDATE to PubSub when nonces are created or reused in
getSSOAuthInfo(), with the network's frontend as a message attribute so
only the correct CV frontend receives it. Listen for ZT1_AUTH_UPDATE
messages and update sso_expiry.authentication_expiry_time accordingly,
with a network existence check before applying.
- Add sso_send_topic/sso_recv_topic to PubSubConfig
- Add PubSubWriter::publishSSONonceUpdate() with frontend param
- Add PubSubSSOListener class for inbound auth updates
- Rename CV1_AUTH_UPDATE to ZT1_AUTH_UPDATE in sso.proto
- Fix pre-existing connection pool leak in getSSOAuthInfo() catch block
Required to get the list of configs for a particular org that the controller has. Named it `linked_id` rather than `org_id` since we don't know what it will be linked to in CV2
Also for pubsub trips back to CV1/CV2, rather than having 2 queues for each service (networks & members), theres now only a single queue for each change type, and `frontend = (cv1|cv2)` attribute is set on the message for filtering.
Muuuuch easier to use external dependencies now
Also tried out conan and vcpkg. Ran into dependency issues when solving for packages to install with conan.
vcpkg is just obtuse as all hell to install and not easy to integrate
PubSub allows us to do schema validation, however it only allows one top level message at a time. Move other sub-message declarations under the main message declaration so that we can enable schema validation in the pubsub stream directly
{"name":"linux-controller-release","inherits":"linux-base","displayName":"Linux Central Controller (Release)","cacheVariables":{"ZT1_CENTRAL_CONTROLLER":"ON"}},
{"name":"linux-controller-debug","inherits":"linux-base","displayName":"Linux Central Controller (Debug)","cacheVariables":{"ZT1_CENTRAL_CONTROLLER":"ON","CMAKE_BUILD_TYPE":"Debug"}},
{"name":"linux-free-release","inherits":"linux-base","displayName":"Linux daemon, free / no controller (Release)","cacheVariables":{"ZT_NONFREE":"OFF"}},
{"name":"linux-free-debug","inherits":"linux-base","displayName":"Linux daemon, free / no controller (Debug)","cacheVariables":{"ZT_NONFREE":"OFF","CMAKE_BUILD_TYPE":"Debug"}},
{"name":"macos-controller-release","inherits":"macos-base","displayName":"macOS Central Controller (Release)","cacheVariables":{"ZT1_CENTRAL_CONTROLLER":"ON"}},
{"name":"macos-controller-debug","inherits":"macos-base","displayName":"macOS Central Controller (Debug)","cacheVariables":{"ZT1_CENTRAL_CONTROLLER":"ON","CMAKE_BUILD_TYPE":"Debug"}},
{"name":"macos-free-release","inherits":"macos-base","displayName":"macOS daemon, free / no controller (Release)","cacheVariables":{"ZT_NONFREE":"OFF"}},
{"name":"macos-free-debug","inherits":"macos-base","displayName":"macOS daemon, free / no controller (Debug)","cacheVariables":{"ZT_NONFREE":"OFF","CMAKE_BUILD_TYPE":"Debug"}},
The binary is `build/zerotier-one`. (On macOS, append the Homebrew prefixes to
`CMAKE_PREFIX_PATH` if your dependencies come from Homebrew, as the presets do.)
### Testing
Typing `make selftest` will build a *zerotier-selftest* binary which unit tests various internals and reports on a few aspects of the build environment. It's a good idea to try this on novel platforms or architectures.
echo'*** FAILED: ZT_BIGTABLE_PROJECT, ZT_BIGTABLE_INSTANCE, and ZT_BIGTABLE_TABLE environment variables must all be defined to use Bigtable as a controller backend'
exit1
fi
BIGTABLE_CONF=", \"bigtable\": {
\"project_id\": \"${ZT_BIGTABLE_PROJECT}\",
\"instance_id\": \"${ZT_BIGTABLE_INSTANCE}\",
\"table_id\": \"${ZT_BIGTABLE_TABLE}\"
}
"
fi
PUBSUB_CONF=""
if["$ZT_USE_PUBSUB"=="true"];then
if[ -z "$ZT_PUBSUB_PROJECT"];then
echo'*** FAILED: ZT_PUBSUB_PROJECT environment variable must be defined to use PubSub as a controller backend'
echo'*** FAILED: ZT_PUBSUB_MEMBER_CHANGE_RECV_TOPIC, ZT_PUBSUB_MEMBER_CHANGE_SEND_TOPIC, ZT_PUBSUB_NETWORK_CHANGE_RECV_TOPIC, and ZT_PUBSUB_NETWORK_CHANGE_SEND_TOPIC environment variables must all be defined to use PubSub as a controller backend'
echo'*** FAILED: controllers assigned to "all" must define ZT_SSO_REDIRECT_URL_CV1 and ZT_SSO_REDIRECT_URL_CV2; the legacy ZT_SSO_REDIRECT_URL cannot serve both Central versions'
exit1
;;
esac
fi
fi
# Per-frontend Central SSO redirect URLs. SSO-enabled networks are routed to the
# SSO endpoint of the Central version ("cv1" or "cv2") they belong to.
Loaded 20 of 1409 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Grant Limberg
Grant Limberg