1.12.0 merge to main (#2104)

* add note about forceTcpRelay

* Create a sample systemd unit for tcp proxy

* set gitattributes for rust & cargo so hashes dont conflict on Windows

* Revert "set gitattributes for rust & cargo so hashes dont conflict on Windows"

This reverts commit 032dc5c108.

* Turn off autocrlf for rust source

Doesn't appear to play nice well when it comes to git and vendored cargo package hashes

* Fix #1883 (#1886)

Still unknown as to why, but the call to `nc->GetProperties()` can fail
when setting a friendly name on the Windows virtual ethernet adapter.
Ensure that `ncp` is not null before continuing and accessing the device
GUID.

* Don't vendor packages for zeroidc (#1885)

* Added docker environment way to join networks (#1871)

* add StringUtils

* fix headers
use recommended headers and remove unused headers

* move extern "C"
only JNI functions need to be exported

* cleanup

* fix ANDROID-50: RESULT_ERROR_BAD_PARAMETER typo

* fix typo in log message

* fix typos in JNI method signatures

* fix typo

* fix ANDROID-51: fieldName is uninitialized

* fix ANDROID-35: memory leak

* fix missing DeleteLocalRef in loops

* update to use unique error codes

* add GETENV macro

* add LOG_TAG defines

* ANDROID-48: add ZT_jnicache.cpp

* ANDROID-48: use ZT_jnicache.cpp and remove ZT_jnilookup.cpp and ZT_jniarray.cpp

* add Event.fromInt

* add PeerRole.fromInt

* add ResultCode.fromInt

* fix ANDROID-36: issues with ResultCode

* add VirtualNetworkConfigOperation.fromInt

* fix ANDROID-40: VirtualNetworkConfigOperation out-of-sync with ZT_VirtualNetworkConfigOperation enum

* add VirtualNetworkStatus.fromInt

* fix ANDROID-37: VirtualNetworkStatus out-of-sync with ZT_VirtualNetworkStatus enum

* add VirtualNetworkType.fromInt

* make NodeStatus a plain data class

* fix ANDROID-52: synchronization bug with nodeMap

* Node init work: separate Node construction and init

* add Node.toString

* make PeerPhysicalPath a plain data class

* remove unused PeerPhysicalPath.fixed

* add array functions

* make Peer a plain data class

* make Version a plain data class

* fix ANDROID-42: copy/paste error

* fix ANDROID-49: VirtualNetworkConfig.equals is wrong

* reimplement VirtualNetworkConfig.equals

* reimplement VirtualNetworkConfig.compareTo

* add VirtualNetworkConfig.hashCode

* make VirtualNetworkConfig a plain data class

* remove unused VirtualNetworkConfig.enabled

* reimplement VirtualNetworkDNS.equals

* add VirtualNetworkDNS.hashCode

* make VirtualNetworkDNS a plain data class

* reimplement VirtualNetworkRoute.equals

* reimplement VirtualNetworkRoute.compareTo

* reimplement VirtualNetworkRoute.toString

* add VirtualNetworkRoute.hashCode

* make VirtualNetworkRoute a plain data class

* add isSocketAddressEmpty

* add addressPort

* add fromSocketAddressObject

* invert logic in a couple of places and return early

* newInetAddress and newInetSocketAddress work
allow newInetSocketAddress to return NULL if given empty address

* fix ANDROID-38: stack corruption in onSendPacketRequested

* use GETENV macro

* JniRef work
JniRef does not use callbacks struct, so remove
fix NewGlobalRef / DeleteGlobalRef mismatch

* use PRId64 macros

* switch statement work

* comments and logging

* Modifier 'public' is redundant for interface members

* NodeException can be made a checked Exception

* 'NodeException' does not define a 'serialVersionUID' field

* 'finalize()' should not be overridden
this is fine to do because ZeroTierOneService calls close() when it is done

* error handling, error reporting, asserts, logging

* simplify loadLibrary

* rename Node.networks -> Node.networkConfigs

* Windows file permissions fix (#1887)

* Allow macOS interfaces to use multiple IP addresses (#1879)

Co-authored-by: Sean OMeara <someara@users.noreply.github.com>
Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* Fix condition where full HELLOs might not be sent when necessary (#1877)

Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* 1.10.4 version bumps

* Add security policy to repo (#1889)

* [+] add e2k64 arch (#1890)

* temp fix for ANDROID-56: crash inside newNetworkConfig from too many args

* 1.10.4 release notes

* Windows 1.10.4 Advanced Installer bump

* Revert "temp fix for ANDROID-56: crash inside newNetworkConfig from too many args"

This reverts commit dd627cd7f4.

* actual fix for ANDROID-56: crash inside newNetworkConfig
cast all arguments to varargs functions as good style

* Fix addIp being called with applied ips (#1897)

This was getting called outside of the check for existing ips
Because of the added ifdef and a brace getting moved to the
wrong place.

```
if (! n.tap()->addIp(*ip)) {
	fprintf(stderr, "ERROR: unable to add ip address %s" ZT_EOL_S, ip->toString(ipbuf));
}
WinFWHelper::newICMPRule(*ip, n.config().nwid);

```

* 1.10.5 (#1905)

* 1.10.5 bump

* 1.10.5 for Windows

* 1.10.5

* Prevent path-learning loops (#1914)

* Prevent path-learning loops

* Only allow new overwrite if not bonded

* fix binding temporary ipv6 addresses on macos (#1910)

The check code wasn't running.

I don't know why !defined(TARGET_OS_IOS) would exclude code on
desktop macOS. I did a quick search and changed it to defined(TARGET_OS_MAC).
Not 100% sure what the most correct solution there is.

You can verify the old and new versions with

`ifconfig | grep temporary`

plus

`zerotier-cli info -j` -> listeningOn

* 1.10.6 (#1929)

* 1.10.5 bump

* 1.10.6

* 1.10.6 AIP for Windows.

* Release notes for 1.10.6 (#1931)

* Minor tweak to Synology Docker image script (#1936)

* Change if_def again so ios can build (#1937)

All apple's variables are "defined"
but sometimes they are defined as "0"

* move begin/commit into try/catch block (#1932)

Thread was exiting in some cases

* Bump openssl from 0.10.45 to 0.10.48 in /zeroidc (#1938)

Bumps [openssl](https://github.com/sfackler/rust-openssl) from 0.10.45 to 0.10.48.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-v0.10.45...openssl-v0.10.48)

---
updated-dependencies:
- dependency-name: openssl
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* new drone bits

* Fix multiple network join from environment entrypoint.sh.release (#1961)

* _bond_m guards _bond, not _paths_m (#1965)

* Fix: warning: mutex '_aqm_m' is not held on every path through here [-Wthread-safety-analysis] (#1964)

* Bump h2 from 0.3.16 to 0.3.17 in /zeroidc (#1963)

Bumps [h2](https://github.com/hyperium/h2) from 0.3.16 to 0.3.17.
- [Release notes](https://github.com/hyperium/h2/releases)
- [Changelog](https://github.com/hyperium/h2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/h2/compare/v0.3.16...v0.3.17)

---
updated-dependencies:
- dependency-name: h2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* Add note that binutils is required on FreeBSD (#1968)

* Add prometheus metrics for Central controllers (#1969)

* add header-only prometheus lib to ext

* rename folder

* Undo rename directory

* prometheus simpleapi included on mac & linux

* wip

* wire up some controller stats

* Get windows building with prometheus

* bsd build flags for prometheus

* Fix multiple network join from environment entrypoint.sh.release (#1961)

* _bond_m guards _bond, not _paths_m (#1965)

* Fix: warning: mutex '_aqm_m' is not held on every path through here [-Wthread-safety-analysis] (#1964)

* Serve prom metrics from /metrics endpoint

* Add prom metrics for Central controller specific things

* reorganize metric initialization

* testing out a labled gauge on Networks

* increment error counter on throw

* Consolidate metrics definitions

Put all metric definitions into node/Metrics.hpp.  Accessed as needed
from there.

* Revert "testing out a labled gauge on Networks"

This reverts commit 499ed6d95e.

* still blows up but adding to the record for completeness right now

* Fix runtime issues with metrics

* Add metrics files to visual studio project

* Missed an "extern"

* add copyright headers to new files

* Add metrics for sent/received bytes (total)

* put /metrics endpoint behind auth

* sendto returns int on Win32

---------

Co-authored-by: Leonardo Amaral <leleobhz@users.noreply.github.com>
Co-authored-by: Brenton Bostick <bostick@gmail.com>

* Central startup update (#1973)

* allow specifying authtoken in central startup

* set allowManagedFrom

* move redis_mem_notification to the correct place

* add node checkins metric

* wire up min/max connection pool size metrics

* x86_64-unknown-linux-gnu on ubuntu runner (#1975)

* adding incoming zt packet type metrics (#1976)

* use cpp-httplib for HTTP control plane (#1979)

refactored the old control plane code to use [cpp-httplib](https://github.com/yhirose/cpp-httplib) instead of a hand rolled HTTP server.  Makes the control plane code much more legible.  Also no longer randomly stops responding.

* Outgoing Packet Metrics (#1980)

add tx/rx labels to packet counters and add metrics for outgoing packets

* Add short-term validation test workflow (#1974)

Add short-term validation test workflow

* Brenton/curly braces (#1971)

* fix formatting

* properly adjust various lines
breakup multiple statements onto multiple lines

* insert {} around if, for, etc.

* Fix rust dependency caching (#1983)

* fun with rust caching

* kick

* comment out invalid yaml keys for now

* Caching should now work

* re-add/rename key directives

* bump

* bump

* bump

* Don't force rebuild on Windows build GH Action (#1985)

Switching `/t:ZeroTierOne:Rebuild` to just `/t:ZeroTierOne` allows the Windows build to use the rust cache.  `/t:ZeroTierOne:Rebuild` cleared the cache before building.

* More packet metrics (#1982)

* found path negotation sends that weren't accounted for

* Fix histogram so it will actually compile

* Found more places for packet metrics

* separate the bind & listen calls on the http backplane (#1988)

* fix memory leak (#1992)

* fix a couple of metrics (#1989)

* More aggressive CLI spamming (#1993)

* fix type signatures (#1991)

* Network-metrics (#1994)

* Add a couple quick functions for converting a uint64_t network ID/node ID into std::string

* Network metrics

* Peer metrics (#1995)

* Adding peer metrics

still need to be wired up for use

* per peer packet metrics

* Fix crash from bad instantiation of histogram

* separate alive & dead path counts

* Add peer metric update block

* add peer latency values in doPingAndKeepalive

* prevent deadlock

* peer latency histogram actually works now

* cleanup

* capture counts of packets to specific peers

---------

Co-authored-by: Joseph Henry <joseph.henry@zerotier.com>

* Metrics consolidation (#1997)

* Rename zt_packet_incoming -> zt_packet

Also consolidate zt_peer_packets into a single metric with tx and rx labels.  Same for ztc_tcp_data and ztc_udp_data

* Further collapse tcp & udp into metric labels for zt_data

* Fix zt_data metric description

* zt_peer_packets description fix

* Consolidate incoming/outgoing network packets to a single metric

* zt_incoming_packet_error -> zt_packet_error

* Disable peer metrics for central controllers

Can change in the future if needed, but given the traffic our controllers serve, that's going to be a *lot* of data

* Disable peer metrics for controllers pt 2

* Update readme files for metrics (#2000)

* Controller Metrics & Network Config Request Fix (#2003)

* add new metrics for network config request queue size and sso expirations
* move sso expiration to its own thread in the controller
* fix potential undefined behavior when modifying a set

* Enable RTTI in Windows build

The new prometheus histogram stuff needs it.

Access violation - no RTTI data!INVALID packet 636ebd9ee8cac6c0 from cafe9efeb9(2605:9880:200:1200:30:571:e34:51/9993) (unexpected exception in tryDecode())

* Don't re-apply routes on BSD

See issue #1986

* Capture setContent by-value instead of by-reference (#2006)

Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* fix typos (#2010)

* central controller metrics & request path updates (#2012)

* internal db metrics

* use shared mutexes for read/write locks

* remove this lock. only used for a metric

* more metrics

* remove exploratory metrics

place controller request benchmarks behind ifdef

* Improve validation test (#2013)

* fix init order for EmbeddedNetworkController (#2014)

* add constant for getifaddrs cache time

* cache getifaddrs - mac

* cache getifaddrs - linux

* cache getifaddrs - bsd

* cache getifaddrs - windows

* Fix oidc client lookup query

join condition referenced the wrong table.  Worked fine unless there were multiple identical client IDs

* Fix udp sent metric

was only incrementing by 1 for each packet sent

* Allow sending all surface addresses to peer in low-bandwidth mode

* allow enabling of low bandwidth mode on controllers

* don't unborrow bad connections

pool will clean them up later

* Multi-arch controller container (#2037)

create arm64 & amd64 images for central controller

* Update README.md

issue #2009

* docker tags change

* fix oidc auth url memory leak (#2031)

getAuthURL() was not calling zeroidc::free_cstr(url);

the only place authAuthURL is called, the url can be retrieved
from the network config instead.

You could alternatively copy the string and call free_cstr in getAuthURL.
If that's better we can change the PR.

Since now there are no callers of getAuthURL I deleted it.

Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* Bump openssl from 0.10.48 to 0.10.55 in /zeroidc (#2034)

Bumps [openssl](https://github.com/sfackler/rust-openssl) from 0.10.48 to 0.10.55.
- [Release notes](https://github.com/sfackler/rust-openssl/releases)
- [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-v0.10.48...openssl-v0.10.55)

---
updated-dependencies:
- dependency-name: openssl
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* zeroidc cargo warnings (#2029)

* fix unused struct member cargo warning

* fix unused import cargo warning

* fix unused return value cargo warning

---------

Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* fix memory leak in macos ipv6/dns helper (#2030)

Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>

* Consider ZEROTIER_JOIN_NETWORKS in healthcheck (#1978)

* Add a 2nd auth token only for access to /metrics (#2043)

* Add a 2nd auth token for /metrics

Allows administrators to distribute a token that only has access to read
metrics and nothing else.

Also added support for using bearer auth tokens for both types of tokens

Separate endpoint for metrics #2041

* Update readme

* fix a couple of cases of writing the wrong token

* Add warning to cli for allow default on FreeBSD

It doesn't work.
Not possible to fix with deficient network
stack and APIs.

ZeroTierOne-freebsd # zerotier-cli set 9bee8941b5xxxxxx allowDefault=1
400 set Allow Default does not work properly on FreeBSD. See #580
root@freebsd13-a:~/ZeroTierOne-freebsd # zerotier-cli get 9bee8941b5xxxxxx allowDefault
1

* ARM64 Support for TapDriver6 (#1949)

* Release memory previously allocated by UPNP_GetValidIGD

* Fix ifdef that breaks libzt on iOS (#2050)

* less drone (#2060)

* Exit if loading an invalid identity from disk (#2058)

* Exit if loading an invalid identity from disk

Previously, if an invalid identity was loaded from disk, ZeroTier would
generate a new identity & chug along and generate a brand new identity
as if nothing happened.  When running in containers, this introduces the
possibility for key matter loss; especially when running in containers
where the identity files are mounted in the container read only.  In
this case, ZT will continue chugging along with a brand new identity
with no possibility of recovering the private key.

ZeroTier should exit upon loading of invalid identity.public/identity.secret #2056

* add validation test for #2056

* tcp-proxy: fix build

* Adjust tcp-proxy makefile to support metrics

There's no way to get the metrics yet. Someone will
have to add the http service.

* remove ZT_NO_METRIC ifdef

* Implement recvmmsg() for Linux to reduce syscalls. (#2046)

Between 5% and 40% speed improvement on Linux, depending on system configuration and load.

* suppress warnings: comparison of integers of different signs: 'int64_t' (aka 'long') and 'uint64_t' (aka 'unsigned long') [-Wsign-compare] (#2063)

* fix warning: 'OS_STRING' macro redefined [-Wmacro-redefined] (#2064)

Even though this is in ext, these particular chunks of code were added
by us, so are ok to modify.

* Apply default route a different way - macOS

The original way we applied default route, by forking
0.0.0.0/0 into 0/1 and 128/1 works, but if mac os has any networking
hiccups -if you change SSIDs or sleep/wake- macos erases the system default route.
And then all networking on the computer is broken.

to summarize the new way:
allowDefault=1
```
sudo route delete default 192.168.82.1
sudo route add default 10.2.0.2
sudo route add -ifscope en1 default 192.168.82.1
```

gives us this routing table
```
Destination        Gateway            RT_IFA             Flags        Refs      Use    Mtu          Netif Expire    rtt(ms) rttvar(ms)
default            10.2.0.2           10.2.0.18          UGScg          90        1   2800       feth4823
default            192.168.82.1       192.168.82.217     UGScIg
```

allowDefault=0
```
sudo route delete default
sudo route delete -ifscope en1 default
sudo route add default 192.168.82.1
```

Notice the I flag, for -ifscope, on the physical default route.

route change does not seem to work reliably.

* fix docker tag for controllers (#2066)

* Update build.sh (#2068)

fix mkwork compilation errors

* Fix network DNS on macOS

It stopped working for ipv4 only networks in Monterey.
See #1696

We add some config like so to System Configuration

```
scutil
show State:/Network/Service/9bee8941b5xxxxxx/IPv4
<dictionary> {
  Addresses : <array> {
    0 : 10.2.1.36
  }
  InterfaceName : feth4823
  Router : 10.2.1.36
  ServerAddress : 127.0.0.1
}

```

* Add search domain to macos dns configuration

Stumbled upon this while debugging something else.
If we add search domain to our system configuration for
network DNS, then search domains work:

```
ping server1                                                                                                                                                                                    ~
PING server1.my.domain (10.123.3.1): 56 data bytes
64 bytes from 10.123.3.1
```

* Fix reporting of secondaryPort and tertiaryPort See: #2039

* Fix typos (#2075)

* Disable executable stacks on assembly objects (#2071)

Add `--noexecstack` to the assembler flags so the resulting binary
will link with a non-executable stack.

Fixes zerotier/ZeroTierOne#1179

Co-authored-by: Joseph Henry <joseph.henry@zerotier.com>

* Test that starting zerotier before internet works

* Don't skip hellos when there are no paths available

working on #2082

* Update validate-1m-linux.sh

* Save zt node log files on abort

* Separate test and summary step in validator script

* Don't apply default route until zerotier is "online"

I was running into issues with restarting the zerotier service while
"full tunnel" mode is enabled.
When zerotier first boots, it gets network state from the cache
on disk. So it immediately applies all the routes it knew about
before it shutdown.
The network config may have change in this time.
If it has, then your default route is via a route
you are blocked from talking on. So you  can't get the current
network config, so your internet does not work.

Other options include
- don't use cached network state on boot
- find a better criteria than "online"

* Fix node time-to-online counter in validator script

* Export variables so that they are accessible by exit function

* Fix PortMapper issue on ZeroTier startup

See issue #2082

We use a call to libnatpmp::ininatpp to make sure the computer
has working network sockets before we go into the main
nat-pmp/upnp logic.

With basic exponenetial delay up to 30 seconds.

* testing

* Comment out PortMapper debug

this got left turned on in a confusing merge previously

* fix macos default route again

see commit fb6af1971 * Fix network DNS on macOS
adding that stuff to System Config causes this extra route to be added
which breaks ipv4 default route.
We figured out a weird System Coniguration setting
that works.

--- old
couldn't figure out how to fix it in SystemConfiguration
so here we are# Please enter the commit message for your changes. Lines starting

We also moved the dns setter to before the syncIps stuff
to help with a race condition. It didn't always work when
you re-joined a network with default route enabled.

* Catch all conditions in switch statement, remove trailing whitespaces

* Add setmtu command, fix bond lifetime issue

* Basic cleanups

* Check if null is passed to VirtualNetworkConfig.equals and name fixes

* ANDROID-96: Simplify and use return code from node_init directly

* Windows arm64 (#2099)

* ARM64 changes for 1.12

* 1.12 Windows advanced installer updates and updates for ARM64

* 1.12.0

* Linux build fixes for old distros.

* release notes

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: travis laduke <travisladuke@gmail.com>
Co-authored-by: Grant Limberg <grant.limberg@zerotier.com>
Co-authored-by: Grant Limberg <glimberg@users.noreply.github.com>
Co-authored-by: Leonardo Amaral <leleobhz@users.noreply.github.com>
Co-authored-by: Brenton Bostick <bostick@gmail.com>
Co-authored-by: Sean OMeara <someara@users.noreply.github.com>
Co-authored-by: Joseph Henry <joseph-henry@users.noreply.github.com>
Co-authored-by: Roman Peshkichev <roman.peshkichev@gmail.com>
Co-authored-by: Joseph Henry <joseph.henry@zerotier.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Stavros Kois <47820033+stavros-k@users.noreply.github.com>
Co-authored-by: Jake Vis <jakevis@outlook.com>
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
Co-authored-by: lison <imlison@foxmail.com>
Co-authored-by: Kenny MacDermid <kenny@macdermid.ca>

.drone.jsonnet

@@ -1,92 +1,256 @@
+//
+// tweakables
+//
+
+local registry = "084037375216.dkr.ecr.us-east-2.amazonaws.com";
+local build_channel = "zerotier-builds";
+local release_channel = "zerotier-releases";
 
 local targets = [
-      //
-      // Render these into .drone.yaml by running "make drone"
-      //
-      { "os": "linux", "name": "el9", "isas": [ "amd64", "arm64", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "el8", "isas": [ "amd64", "arm64", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "el7", "isas": [ "amd64", "ppc64le"], "events": [ "tag" ] },
-      { "os": "linux", "name": "el6", "isas": [ "amd64" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "amzn2", "isas": [ "amd64", "arm64" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "fc37", "isas": [ "amd64", "arm64", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "fc36", "isas": [ "amd64", "arm64", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "fc35", "isas": [ "amd64", "arm64", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "jammy", "isas": [ "amd64", "arm64", "armv7", "riscv64", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "focal", "isas": [ "amd64", "arm64", "armv7", "riscv64", "ppc64le"  ], "events": [ "tag" ] },
-      { "os": "linux", "name": "bionic", "isas": [ "amd64", "arm64", "386", "ppc64le", "s390x" ], "events": ["tag" ] },
-      { "os": "linux", "name": "xenial", "isas": [ "amd64", "arm64", "386" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "sid", "isas": [ "386", "amd64", "arm64",  "riscv64", "mips64le", "ppc64le", "s390x" ], "events":   [ "push", "tag" ] },
-      { "os": "linux", "name": "bookworm", "isas": [ "amd64", "arm64", "armv7", "386", "mips64le", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "bullseye", "isas": [ "amd64", "arm64", "armv7", "386", "mips64le", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "buster", "isas": [ "amd64", "arm64", "armv7", "386", "mips64le", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "stretch", "isas": [ "amd64", "arm64", "386" ], "events": [ "tag" ] },
-      // { "os": "windows", "name": "win2k19", "isas": [ "amd64" ], "events": ["push", "tag" ] }
+    { "os": "linux", distro: "redhat", "name": "el9",      "isas": [                 "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "tag", "custom" ] },
+    { "os": "linux", distro: "redhat", "name": "el8",      "isas": [                 "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "tag" ] },
+    { "os": "linux", distro: "redhat", "name": "el7",      "isas": [ "386",          "amd64",          "ppc64le"],                                  "events": [ "tag" ] },
+    { "os": "linux", distro: "amazon", "name": "amzn2",    "isas": [                 "amd64", "arm64" ],                                            "events": [ "tag" ] },
+    { "os": "linux", distro: "amazon", "name": "amzn2022", "isas": [                 "amd64", "arm64" ],                                            "events": [ "tag" ] },
+    { "os": "linux", distro: "fedora", "name": "fc38",     "isas": [                 "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "tag" ] },
+    { "os": "linux", distro: "fedora", "name": "fc37",     "isas": [                 "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "tag" ] },
+    { "os": "linux", distro: "fedora", "name": "fc36",     "isas": [                 "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "tag" ] },
+    { "os": "linux", distro: "ubuntu", "name": "jammy",    "isas": [        "armv7", "amd64", "arm64", "ppc64le", "s390x", "riscv64" ],             "events": [ "tag" ] },
+    { "os": "linux", distro: "ubuntu", "name": "focal",    "isas": [        "armv7", "amd64", "arm64", "ppc64le", "s390x", "riscv64" ],             "events": [ "tag" ] },
+    { "os": "linux", distro: "ubuntu", "name": "bionic",   "isas": [ "386", "armv7", "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "tag" ] },
+    { "os": "linux", distro: "ubuntu", "name": "xenial",   "isas": [ "386", "armv7", "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "tag" ] },
+    { "os": "linux", distro: "ubuntu", "name": "trusty",   "isas": [ "386", "armv7", "amd64", "arm64" ],                                            "events": [ "tag" ] },
+    { "os": "linux", distro: "debian", "name": "bookworm", "isas": [ "386", "armv7", "amd64", "arm64", "mips64le", "ppc64le", "s390x" ],            "events": [ "tag"] },
+    { "os": "linux", distro: "debian", "name": "bullseye", "isas": [ "386", "armv7", "amd64", "arm64", "mips64le", "ppc64le", "s390x" ],            "events": [ "push", "tag", "custom" ] },
+    { "os": "linux", distro: "debian", "name": "buster",   "isas": [ "386", "armv7", "amd64", "arm64" ],                                            "events": [ "tag" ] },
+    { "os": "linux", distro: "debian", "name": "stretch",  "isas": [ "386", "armv7", "amd64", "arm64" ],                                            "events": [ "tag" ] },
+    { "os": "linux", distro: "debian", "name": "jessie",   "isas": [ "386", "armv7", "amd64" ],                                                     "events": [ "tag" ] },
+
+//   { "os": "windows", distro: "windows", "name": "windows",  "isas": [ "amd64" ], "events": [ "push", "tag", "custom" ] },
+//   { "os": "darwin", distro: "darwin", "name": "darwin",  "isas": [ "amd64" ], "events": [ "push", "tag", "custom" ] },
+
 ];
 
+local less_targets = [
+      { "os": "linux", distro: "redhat", "name": "el9",      "isas": [                 "amd64", "arm64" ],                        "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "redhat", "name": "el8",      "isas": [                 "amd64", "arm64" ],                        "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "ubuntu", "name": "jammy",    "isas": [        "armv7", "amd64", "arm64" ],             "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "ubuntu", "name": "focal",    "isas": [        "armv7", "amd64", "arm64" ],             "events": [ "push", "tag", "custom" ] },
+];
+
+
+local native_targets = [
+      { "os": "linux", distro: "debian", "name": "bullseye", "isas": [ "386", "armv7", "amd64", "arm64" ],            "events": [ "push", "tag", "custom" ] },
+];    
+
 local master_targets = [
       //
-      // Render these into .drone.yaml by running "make drone"
-      //      
-      { "os": "linux", "name": "el9", "isas": [ "amd64", "arm64", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "el8", "isas": [ "amd64", "arm64", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "el7", "isas": [ "amd64", "ppc64le"], "events": [ "tag" ] },
-      { "os": "linux", "name": "el6", "isas": [ "amd64" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "amzn2", "isas": [ "amd64", "arm64" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "fc37", "isas": [ "amd64", "arm64", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "fc36", "isas": [ "amd64", "arm64", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "fc35", "isas": [ "amd64", "arm64", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "jammy", "isas": [ "amd64", "arm64", "armv7", "riscv64", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "focal", "isas": [ "amd64", "arm64", "armv7", "riscv64", "ppc64le"  ], "events": [ "tag" ] },
-      { "os": "linux", "name": "bionic", "isas": [ "amd64", "arm64", "386", "ppc64le", "s390x" ], "events": ["tag" ] },
-      { "os": "linux", "name": "xenial", "isas": [ "amd64", "arm64", "386" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "sid", "isas": [ "386", "amd64", "arm64",  "riscv64", "mips64le", "ppc64le", "s390x" ], "events": [ "push", "tag" ] },
-      { "os": "linux", "name": "bookworm", "isas": [ "amd64", "arm64", "armv7", "386", "mips64le", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "bullseye", "isas": [ "amd64", "arm64", "armv7", "386", "mips64le", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "buster", "isas": [ "amd64", "arm64", "armv7", "386", "mips64le", "ppc64le", "s390x" ], "events": [ "tag" ] },
-      { "os": "linux", "name": "stretch", "isas": [ "amd64", "arm64", "386" ], "events": [ "tag" ] },
-      // { "os": "windows", "name": "win2k19", "isas": [ "amd64" ], "events": ["push", "tag" ] }
+      // copypasta from here
+      //
+      { "os": "linux", distro: "redhat", "name": "el9",      "isas": [                 "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "redhat", "name": "el8",      "isas": [                 "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "redhat", "name": "el7",      "isas": [ "386", "amd64",          "ppc64le"],                                  "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "amazon", "name": "amzn2",    "isas": [                 "amd64", "arm64" ],                                            "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "amazon", "name": "amzn2022", "isas": [                 "amd64", "arm64" ],                                            "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "fedora", "name": "fc38",     "isas": [                 "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "fedora", "name": "fc37",     "isas": [                 "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "fedora", "name": "fc36",     "isas": [                 "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "ubuntu", "name": "jammy",    "isas": [        "armv7", "amd64", "arm64", "ppc64le", "s390x", "riscv64" ],             "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "ubuntu", "name": "focal",    "isas": [        "armv7", "amd64", "arm64", "ppc64le", "s390x", "riscv64" ],             "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "ubuntu", "name": "bionic",   "isas": [ "386", "armv7", "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "ubuntu", "name": "xenial",   "isas": [ "386", "armv7", "amd64", "arm64", "ppc64le", "s390x" ],                        "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "ubuntu", "name": "trusty",   "isas": [ "386", "armv7", "amd64", "arm64" ],                                            "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "debian", "name": "sid",      "isas": [ "386", "armv7", "amd64", "arm64", "mips64le", "ppc64le", "s390x", "riscv64" ], "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "debian", "name": "bookworm", "isas": [ "386", "armv7", "amd64", "arm64", "mips64le", "ppc64le", "s390x" ],            "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "debian", "name": "bullseye", "isas": [ "386", "armv7", "amd64", "arm64", "mips64le", "ppc64le", "s390x" ],            "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "debian", "name": "buster",   "isas": [ "386", "armv7", "amd64", "arm64" ],                                            "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "debian", "name": "stretch",  "isas": [ "386", "armv7", "amd64", "arm64" ],                                            "events": [ "push", "tag", "custom" ] },
+      { "os": "linux", distro: "debian", "name": "jessie",   "isas": [ "386", "armv7", "amd64" ],                                                     "events": [ "push", "tag", "custom" ] },
+      { "os": "windows", distro: "windows", "name": "win2k22", "isas": [ "amd64" ],                                                                     "events": [ "push", "tag", "custom" ] }
 ];
 
-local Build(platform, os, isa, events) = {
+//
+// functions
+//
+
+local pipeline_type(os)  = if os == "darwin" then "exec" else "docker";
+local builder_image(os)  = if os == "linux" then registry + "/honda-builder" else registry + "/windows-builder";
+local tester_image(os)   = if os == "linux" then registry + "/honda-builder" else registry + "/windows-tester";
+local build_step_volumes(os) = if os == "linux" then [ { name: "zerotier-builds", path: "/zerotier-builds" } ] else [];
+local release_step_volumes(os) = if os == "linux" then [ { name: "zerotier-releases", path: "/zerotier-releases" } ] else [];
+local host_volumes(os)   = if os == "linux" then [
+  { name: "zerotier-builds", host: { path: "/zerotier-builds" } },
+  { name: "zerotier-releases", host: { path: "/zerotier-releases" } },
+] else [];
+
+local index_image(distro) =
+      if distro == "debian" || distro == "ubuntu" then
+          registry + "/apt-builder"
+      else if distro == "redhat" || distro == "fedora" || distro == "amazon" then
+          registry + "/dnf-builder"
+      else if distro == "windows" then
+          registry + "/msi-builder"
+;          
+
+local copy_commands(os, distro, name, isa, version) =
+  if os == "linux" then [
+      std.join(" ", [ "./ci/scripts/publish.sh", name, distro, isa, version, "${DRONE_BUILD_EVENT}" ])
+    ]
+    else if os == "windows" then [  
+      "C:\\scripts\\fix-ec2-metadata.ps1",
+      "Get-ChildItem windows",
+      // "aws s3 cp windows\\bytey-SetupFiles\\bytey.msi s3://zerotier-builds/windows/" + version + "/bytey.msi",
+    ] else if os == "darwin" then [
+        "echo hello"
+      ]
+;
+
+local index_commands(os, channel, distro, name, isas) =
+      if os == "linux" then
+        [ "/usr/local/bin/index " + channel + " " + distro + " " + name  + " " + std.join(" ", isas) ]
+      else if os == "windows" then
+        [ "Get-ChildItem -Recurse windows" ]
+;
+
+local build_commands(os, distro, name, isa, version) =
+      if os == "linux" then
+        [ std.join(" ", [ "./ci/scripts/build.sh", name, distro, isa, version, "${DRONE_BUILD_EVENT}" ]) ]
+      else
+        if os == "windows" then
+           [ "windows/build.ps1", "windows/package.ps1" ]
+      else
+        if os == "darwin" then
+           [ "whoami" ]
+;
+
+local test_commands(os, distro, name, isa, version) =
+  if os == "linux" then
+    [ std.join(" ", [ "./ci/scripts/test.sh", name, distro, isa, version, "${DRONE_BUILD_EVENT}" ]) ]
+  else
+    if os == "windows" then
+      [ "windows/testpackage.ps1 " + version ]
+;
+
+//
+// render
+//
+
+local Build(os, distro, name, isa, events) = {
   "kind": "pipeline",
-  "type": "docker",
+  "type": pipeline_type(os),
+  "name": std.join(" ", [ name, isa, "build" ]),
   "pull": "always",
-  "name": platform + " " + isa + " " + "build",
-  "clone": { "depth": 1 },
+  "clone": { "depth": 1, [ if os == "darwin" then "disable" ]: true },
   "steps": [
     {
       "name": "build",
-      "image": "registry.sean.farm/honda-builder",
-      "commands": [ "./ci/scripts/build.sh " + platform + " " + isa + " " + "100.0.0+${DRONE_COMMIT_SHA:0:8}" + " " + "${DRONE_BUILD_EVENT}" ]
+      "image": builder_image(os),
+      "commands": build_commands(os, distro, name, isa, "100.0.0+${DRONE_COMMIT_SHA:0:8}"),
+      "when": { "event": [ "push" ]},
     },
     {
-      "name": "list",
-      "image": "registry.sean.farm/honda-builder",
-      "commands": [ "ls -la " + platform ]
+      "name": "release",
+      "image": builder_image(os),
+      "commands": build_commands(os, distro, name, isa, "${DRONE_TAG}"),
+      "when": { "event": [ "tag" ]},
+    },    
+    {
+      "name": "copy build",
+      "image": builder_image(os),
+      "commands": copy_commands(os, distro, name, isa, "100.0.0+${DRONE_COMMIT_SHA:0:8}"),
+      "volumes": build_step_volumes(os),
+      "when": { "event": [ "push" ]},
     },
-    // {
-    //   "name": "notify-mattermost",
-    //   "image": "registry.sean.farm/mattermost-notify",
-    //   "environment": {
-    //     "token": { "from_secret": "mattermost-token" },
-    //     "host": { "from_secret": "mattermost-host" },
-    //     "channel": { "from_secret": "mattermost-channel" },
-    //     "maxRetry": 3,
-    //   },
-    //   "when": { "status": [ "failure" ] }
-    // }
-  ],  
-  "image_pull_secrets": [ "dockerconfigjson" ],
-  [ if isa == "arm64" || isa == "armv7" then "platform" ]: { os: os, arch: "arm64" },
+    {
+      "name": "copy relase",
+      "image": builder_image(os),
+      "commands": copy_commands(os, distro, name, isa, "${DRONE_TAG}"),
+      "volumes": release_step_volumes(os),
+      "when": { "event": [ "tag" ]},      
+    },    
+  ],
+  "volumes": host_volumes(os),
+  "platform": { "os": os, [ if isa == "arm64" || isa == "armv7" then "arch" ]: "arm64" },
   "trigger": { "event": events }
 };
 
-// puttin on the bits
+local Test(os, distro, name, isa, events) = {
+  "kind": "pipeline",
+  "type": pipeline_type(os),
+  "name": std.join(" ", [ name, isa, "test"]),
+  "pull": "always",
+  "clone": { "depth": 1 },
+  "steps": [
+    {
+      "name": "test build",
+      "image": tester_image(os),
+      "volumes": build_step_volumes(os),
+      "commands": test_commands(os, distro, name, isa, "100.0.0+${DRONE_COMMIT_SHA:0:8}"),
+      "when": { "event": [ "push" ]},
+    },
+    {
+      "name": "test release",
+      "image": tester_image(os),
+      "volumes": release_step_volumes(os),
+      "commands": test_commands(os, distro, name, isa, "${DRONE_TAG}"),
+      "when": { "event": [ "tag" ]},
+    },    
+  ],
+  "volumes": host_volumes(os),
+  "platform": { "os": os, [ if isa == "arm64" || isa == "armv7" then "arch" ]: "arm64" },
+  "depends_on": [ std.join(" ", [ name, "index" ]) ],
+  "trigger": { "event": events }
+};
+
+local Index(p) = {
+  "kind": "pipeline",
+  "type": pipeline_type(p.os),
+  "name": std.join(" ", [ p.name, "index" ]),
+  "pull": "always",
+  "clone": { "depth": 1 },
+  "steps": [
+    {
+      "name": "index build",
+      "image": index_image(p.distro),
+      "commands": index_commands(p.os, "zerotier-builds", p.distro, p.name, p.isas),
+      "volumes": build_step_volumes(p.os),
+      "environment":{ "GPG_PRIVATE_KEY": { from_secret: "gpg-private-key" }},
+      "when": { "event": [ "push" ]},
+    },
+    {
+      "name": "index release",
+      "image": index_image(p.distro),
+      "commands": index_commands(p.os, "zerotier-releases", p.distro, p.name, p.isas),
+      "volumes": release_step_volumes(p.os),
+      "environment":{ "GPG_PRIVATE_KEY": { from_secret: "gpg-private-key" }},
+      "when": { "event": [ "tag" ]},  
+    },    
+  ],
+  "volumes": host_volumes(p.os),
+  "platform": { "os": p.os },
+  depends_on: std.flattenArrays([ [ std.join(" ", [ p.name, isa, "build" ]) ] for isa in p.isas ]),
+  "trigger": { "event": p.events }
+};
+
+//
+// print
+//
 
 std.flattenArrays([
-  [
-     Build(p.name, p.os, isa, p.events)
-      for isa in p.isas
-  ]
-  for p in targets
-])
+    [
+      Build(p.os, p.distro, p.name, isa, p.events)
+        for isa in p.isas
+    ] +
+    [
+      Index(p)
+    ]
+    for p in native_targets
+ ]) +
+ std.flattenArrays([
+     [
+        Test(p.os, p.distro, p.name, isa, p.events)
+         for isa in p.isas
+     ]
+     for p in native_targets
+ ])
+ 

.github/workflows/build.yml

@@ -0,0 +1,98 @@
+on: [ push ]
+
+jobs:
+  build_ubuntu:
+    runs-on: ubuntu-latest
+    steps:
+    - name: gitconfig
+      run: |
+        git config --global core.autocrlf input
+      #        git config --global core.eol lf
+    - name: checkout
+      uses: actions/checkout@v3
+    - name: Install Rust
+      uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+        target: x86_64-unknown-linux-gnu
+        override: true
+        components: rustfmt, clippy
+
+    - name: Set up cargo cache
+      uses: Swatinem/rust-cache@v2
+      continue-on-error: false
+      with:
+        key: ${{ runner.os }}-cargo-${{ hashFiles('zeroidc//Cargo.lock') }}
+        shared-key: ${{ runner.os }}-cargo-
+        workspaces: |
+          zeroidc/
+
+    - name: make
+      run: make
+    - name: selftest
+      run: |
+        make selftest
+        ./zerotier-selftest
+
+  build_macos:
+    runs-on: macos-latest
+    steps:
+    - name: gitconfig
+      run: |
+        git config --global core.autocrlf input
+      #        git config --global core.eol lf
+    - name: checkout
+      uses: actions/checkout@v3
+    - name: Install Rust
+      uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+        target: aarch64-apple-darwin
+        override: true
+        components: rustfmt, clippy
+    - name: Set up cargo cache
+      uses: Swatinem/rust-cache@v2
+      continue-on-error: false
+      with:
+        key: ${{ runner.os }}-cargo-${{ hashFiles('zeroidc//Cargo.lock') }}
+        shared-key: ${{ runner.os }}-cargo-
+        workspaces: |
+          zeroidc/
+
+    - name: make
+      run: make
+    - name: selftest
+      run: |
+        make selftest
+        ./zerotier-selftest
+
+  build_windows:
+    runs-on: windows-latest
+    steps:
+    - name: gitconfig
+      run: |
+        git config --global core.autocrlf true
+      #        git config --global core.eol lf
+    - name: checkout
+      uses: actions/checkout@v3
+    - name: Install Rust
+      uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+        target: aarch64-apple-darwin
+        override: true
+        components: rustfmt, clippy
+    - name: Set up cargo cache
+      uses: Swatinem/rust-cache@v2
+      continue-on-error: false
+      with:
+        key: ${{ runner.os }}-cargo-${{ hashFiles('zeroidc//Cargo.lock') }}
+        shared-key: ${{ runner.os }}-cargo-
+        workspaces: |
+          zeroidc/
+
+    - name: setup msbuild
+      uses: microsoft/setup-msbuild@v1.1.3
+    - name: msbuild
+      run: |
+        msbuild windows\ZeroTierOne.sln /m /p:Configuration=Release  /property:Platform=x64 /t:ZeroTierOne        

.github/workflows/validate-linux.sh

@@ -0,0 +1,461 @@
+#!/bin/bash
+
+# This test script joins Earth and pokes some stuff
+
+TEST_NETWORK=8056c2e21c000001
+RUN_LENGTH=30
+TEST_FINISHED=false
+ZTO_VER=$(git describe --tags $(git rev-list --tags --max-count=1))
+ZTO_COMMIT=$(git rev-parse HEAD)
+ZTO_COMMIT_SHORT=$(git rev-parse --short HEAD)
+TEST_DIR_PREFIX="$ZTO_VER-$ZTO_COMMIT_SHORT-test-results"
+
+TEST_OK=0
+TEST_FAIL=1
+
+echo "Performing test on: $ZTO_VER-$ZTO_COMMIT_SHORT"
+TEST_FILEPATH_PREFIX="$TEST_DIR_PREFIX/$ZTO_COMMIT_SHORT"
+mkdir $TEST_DIR_PREFIX
+
+# How long we will wait for ZT to come online before considering it a failure
+MAX_WAIT_SECS=30
+
+################################################################################
+# Multi-node connectivity and performance test                                 #
+################################################################################
+
+test() {
+
+	echo -e "\nPerforming pre-flight checks"
+
+	check_exit_on_invalid_identity
+
+	echo -e "\nRunning test for $RUN_LENGTH seconds"
+
+	export NS1="ip netns exec ns1"
+	export NS2="ip netns exec ns2"
+
+	export ZT1="$NS1 ./zerotier-cli -p9996 -D$(pwd)/node1"
+	# Specify custom port on one node to ensure that feature works
+	export ZT2="$NS2 ./zerotier-cli -p9997 -D$(pwd)/node2"
+
+	echo -e "\nSetting up network namespaces..."
+	echo "Setting up ns1"
+
+	ip netns add ns1
+	$NS1 ip link set dev lo up
+	ip link add veth0 type veth peer name veth1
+	ip link set veth1 netns ns1
+	ip addr add 192.168.0.1/24 dev veth0
+	ip link set dev veth0 up
+
+	$NS1 ip addr add 192.168.0.2/24 dev veth1
+	$NS1 ip link set dev veth1 up
+
+	# Add default route
+	$NS1 ip route add default via 192.168.0.1
+
+	iptables -t nat -A POSTROUTING -s 192.168.0.0/255.255.255.0 \
+		-o eth0 -j MASQUERADE
+	iptables -A FORWARD -i eth0 -o veth0 -j ACCEPT
+	iptables -A FORWARD -o eth0 -i veth0 -j ACCEPT
+
+	echo "Setting up ns2"
+	ip netns add ns2
+	$NS2 ip link set dev lo up
+	ip link add veth2 type veth peer name veth3
+	ip link set veth3 netns ns2
+	ip addr add 192.168.1.1/24 dev veth2
+	ip link set dev veth2 up
+
+	$NS2 ip addr add 192.168.1.2/24 dev veth3
+	$NS2 ip link set dev veth3 up
+	$NS2 ip route add default via 192.168.1.1
+
+	iptables -t nat -A POSTROUTING -s 192.168.1.0/255.255.255.0 \
+		-o eth0 -j MASQUERADE
+	iptables -A FORWARD -i eth0 -o veth2 -j ACCEPT
+	iptables -A FORWARD -o eth0 -i veth2 -j ACCEPT
+
+	# Allow forwarding
+	sysctl -w net.ipv4.ip_forward=1
+
+	################################################################################
+	# Memory Leak Check                                                            #
+	################################################################################
+
+	export FILENAME_MEMORY_LOG="$TEST_FILEPATH_PREFIX-memory.log"
+
+	echo -e "\nStarting a ZeroTier instance in each namespace..."
+
+	export time_test_start=$(date +%s)
+
+	# Spam the CLI as ZeroTier is starting
+	spam_cli 100
+
+	echo "Starting memory leak check"
+	$NS1 sudo valgrind --demangle=yes --exit-on-first-error=yes \
+		--error-exitcode=1 \
+		--xml=yes \
+		--xml-file=$FILENAME_MEMORY_LOG \
+		--leak-check=full \
+		./zerotier-one node1 -p9996 -U >>node_1.log 2>&1 &
+
+	# Second instance, not run in memory profiler
+	# Don't set up internet access until _after_ zerotier is running
+	# This has been a source of stuckness in the past.
+	$NS2 ip addr del 192.168.1.2/24 dev veth3
+	$NS2 sudo ./zerotier-one node2 -U -p9997 >>node_2.log 2>&1 &
+	sleep 1;
+	$NS2 ip addr add 192.168.1.2/24 dev veth3
+	$NS2 ip route add default via 192.168.1.1
+
+	echo -e "\nPing from host to namespaces"
+
+	ping -c 3 192.168.0.1
+	ping -c 3 192.168.1.1
+
+	echo -e "\nPing from namespace to host"
+
+	$NS1 ping -c 3 192.168.0.1
+	$NS1 ping -c 3 192.168.0.1
+	$NS2 ping -c 3 192.168.0.2
+	$NS2 ping -c 3 192.168.0.2
+
+	echo -e "\nPing from ns1 to ns2"
+
+	$NS1 ping -c 3 192.168.0.1
+
+	echo -e "\nPing from ns2 to ns1"
+
+	$NS2 ping -c 3 192.168.0.1
+
+	################################################################################
+	# Online Check                                                                 #
+	################################################################################
+
+	echo "Waiting for ZeroTier to come online before attempting test..."
+	node1_online=false
+	node2_online=false
+	both_instances_online=false
+	time_zt_node1_start=$(date +%s)
+	time_zt_node2_start=$(date +%s)
+
+	for ((s = 0; s <= $MAX_WAIT_SECS; s++)); do
+		node1_online="$($ZT1 -j info | jq '.online' 2>/dev/null)"
+		node2_online="$($ZT2 -j info | jq '.online' 2>/dev/null)"
+		echo "Checking for online status: try #$s, node1:$node1_online, node2:$node2_online"
+		if [[ "$node2_online" == "true" && "$node1_online" == "true" ]]; then
+			export both_instances_online=true
+			export time_to_both_nodes_online=$(date +%s)
+			break
+		fi
+		sleep 1
+	done
+
+	echo -e "\n\nContents of ZeroTier home paths:"
+
+	ls -lga node1
+	tree node1
+	ls -lga node2
+	tree node2
+
+	echo -e "\n\nRunning ZeroTier processes:"
+	echo -e "\nNode 1:\n"
+	$NS1 ps aux | grep zerotier-one
+	echo -e "\nNode 2:\n"
+	$NS2 ps aux | grep zerotier-one
+
+	echo -e "\n\nStatus of each instance:"
+
+	echo -e "\n\nNode 1:\n"
+	$ZT1 status
+	echo -e "\n\nNode 2:\n"
+	$ZT2 status
+
+	if [[ "$both_instances_online" != "true" ]]; then
+		exit_test_and_generate_report $TEST_FAIL "one or more nodes failed to come online"
+	fi
+
+	echo -e "\nJoining networks"
+
+	$ZT1 join $TEST_NETWORK
+	$ZT2 join $TEST_NETWORK
+
+	sleep 10
+
+	node1_ip4=$($ZT1 get $TEST_NETWORK ip4)
+	node2_ip4=$($ZT2 get $TEST_NETWORK ip4)
+
+	echo "node1_ip4=$node1_ip4"
+	echo "node2_ip4=$node2_ip4"
+
+	echo -e "\nPinging each node"
+
+	PING12_FILENAME="$TEST_FILEPATH_PREFIX-ping-1-to-2.txt"
+	PING21_FILENAME="$TEST_FILEPATH_PREFIX-ping-2-to-1.txt"
+
+	$NS1 ping -c 16 $node2_ip4 >$PING12_FILENAME
+	$NS2 ping -c 16 $node1_ip4 >$PING21_FILENAME
+
+	ping_loss_percent_1_to_2=$(cat $PING12_FILENAME |
+		grep "packet loss" | awk '{print $6}' | sed 's/%//')
+	ping_loss_percent_2_to_1=$(cat $PING21_FILENAME |
+		grep "packet loss" | awk '{print $6}' | sed 's/%//')
+
+	# Normalize loss value
+	export ping_loss_percent_1_to_2=$(echo "scale=2; $ping_loss_percent_1_to_2/100.0" | bc)
+	export ping_loss_percent_2_to_1=$(echo "scale=2; $ping_loss_percent_2_to_1/100.0" | bc)
+
+	################################################################################
+	# CLI Check                                                                    #
+	################################################################################
+
+	echo "Testing basic CLI functionality..."
+
+	spam_cli 10
+
+	$ZT1 join $TEST_NETWORK
+
+	$ZT1 -h
+	$ZT1 -v
+	$ZT1 status
+	$ZT1 info
+	$ZT1 listnetworks
+	$ZT1 peers
+	$ZT1 listpeers
+
+	$ZT1 -j status
+	$ZT1 -j info
+	$ZT1 -j listnetworks
+	$ZT1 -j peers
+	$ZT1 -j listpeers
+
+	$ZT1 dump
+
+	$ZT1 get $TEST_NETWORK allowDNS
+	$ZT1 get $TEST_NETWORK allowDefault
+	$ZT1 get $TEST_NETWORK allowGlobal
+	$ZT1 get $TEST_NETWORK allowManaged
+	$ZT1 get $TEST_NETWORK bridge
+	$ZT1 get $TEST_NETWORK broadcastEnabled
+	$ZT1 get $TEST_NETWORK dhcp
+	$ZT1 get $TEST_NETWORK id
+	$ZT1 get $TEST_NETWORK mac
+	$ZT1 get $TEST_NETWORK mtu
+	$ZT1 get $TEST_NETWORK name
+	$ZT1 get $TEST_NETWORK netconfRevision
+	$ZT1 get $TEST_NETWORK nwid
+	$ZT1 get $TEST_NETWORK portDeviceName
+	$ZT1 get $TEST_NETWORK portError
+	$ZT1 get $TEST_NETWORK status
+	$ZT1 get $TEST_NETWORK type
+
+	# Test an invalid command
+	$ZT1 get $TEST_NETWORK derpderp
+
+	# TODO: Validate JSON
+
+	# Performance Test
+
+	export FILENAME_PERF_JSON="$TEST_FILEPATH_PREFIX-iperf.json"
+
+	echo -e "\nBeginning performance test:"
+
+	echo -e "\nStarting server:"
+
+	echo "$NS1 iperf3 -s &"
+	sleep 1
+
+	echo -e "\nStarting client:"
+	sleep 1
+
+	echo "$NS2 iperf3 --json -c $node1_ip4 > $FILENAME_PERF_JSON"
+
+	cat $FILENAME_PERF_JSON
+
+	# Let ZeroTier idle long enough for various timers
+
+	echo -e "\nIdling ZeroTier for $RUN_LENGTH seconds..."
+	sleep $RUN_LENGTH
+
+	echo -e "\nLeaving networks"
+
+	$ZT1 leave $TEST_NETWORK
+	$ZT2 leave $TEST_NETWORK
+
+	sleep 5
+
+	exit_test_and_generate_report $TEST_OK "completed test"
+}
+
+################################################################################
+# Generate report                                                              #
+################################################################################
+
+exit_test_and_generate_report() {
+
+	echo -e "\nStopping memory check..."
+	sudo pkill -15 -f valgrind
+	sleep 10
+
+	time_test_end=$(date +%s)
+
+	echo "Exiting test with reason: $2 ($1)"
+
+	# Collect ZeroTier dump files
+
+	echo -e "\nCollecting ZeroTier dump files"
+
+	node1_id=$($ZT1 -j status | jq -r .address)
+	node2_id=$($ZT2 -j status | jq -r .address)
+
+	$ZT1 dump
+	mv zerotier_dump.txt "$TEST_FILEPATH_PREFIX-node-dump-$node1_id.txt"
+
+	$ZT2 dump
+	mv zerotier_dump.txt "$TEST_FILEPATH_PREFIX-node-dump-$node2_id.txt"
+
+	# Copy ZeroTier stdout/stderr logs
+
+	cp node_1.log "$TEST_FILEPATH_PREFIX-node-log-$node1_id.txt"
+	cp node_2.log "$TEST_FILEPATH_PREFIX-node-log-$node2_id.txt"
+
+	# Generate report
+
+	cat $FILENAME_MEMORY_LOG
+
+	DEFINITELY_LOST=$(xmlstarlet sel -t -v '/valgrindoutput/error/xwhat' \
+		$FILENAME_MEMORY_LOG | grep "definitely" | awk '{print $1;}')
+	POSSIBLY_LOST=$(xmlstarlet sel -t -v '/valgrindoutput/error/xwhat' \
+		$FILENAME_MEMORY_LOG | grep "possibly" | awk '{print $1;}')
+
+	# Generate coverage report artifact and summary
+
+	FILENAME_COVERAGE_JSON="$TEST_FILEPATH_PREFIX-coverage.json"
+	FILENAME_COVERAGE_HTML="$TEST_FILEPATH_PREFIX-coverage.html"
+
+	echo -e "\nGenerating coverage test report..."
+
+	gcovr -r . --exclude ext --json-summary $FILENAME_COVERAGE_JSON \
+		--html >$FILENAME_COVERAGE_HTML
+
+	cat $FILENAME_COVERAGE_JSON
+
+	COVERAGE_LINE_COVERED=$(cat $FILENAME_COVERAGE_JSON | jq .line_covered)
+	COVERAGE_LINE_TOTAL=$(cat $FILENAME_COVERAGE_JSON | jq .line_total)
+	COVERAGE_LINE_PERCENT=$(cat $FILENAME_COVERAGE_JSON | jq .line_percent)
+
+	COVERAGE_LINE_COVERED="${COVERAGE_LINE_COVERED:-0}"
+	COVERAGE_LINE_TOTAL="${COVERAGE_LINE_TOTAL:-0}"
+	COVERAGE_LINE_PERCENT="${COVERAGE_LINE_PERCENT:-0}"
+
+	# Default values
+
+	DEFINITELY_LOST="${DEFINITELY_LOST:-0}"
+	POSSIBLY_LOST="${POSSIBLY_LOST:-0}"
+	ping_loss_percent_1_to_2="${ping_loss_percent_1_to_2:-100.0}"
+	ping_loss_percent_2_to_1="${ping_loss_percent_2_to_1:-100.0}"
+	time_to_both_nodes_online="${time_to_both_nodes_online:--1}"
+
+	# Summarize and emit json for trend reporting
+
+	FILENAME_SUMMARY="$TEST_FILEPATH_PREFIX-summary.json"
+
+	time_length_test=$((time_test_end - time_test_start))
+	if [[ $time_to_both_nodes_online != -1 ]];
+	then
+		time_to_both_nodes_online=$((time_to_both_nodes_online - time_test_start))
+	fi
+	#time_length_zt_join=$((time_zt_join_end-time_zt_join_start))
+	#time_length_zt_leave=$((time_zt_leave_end-time_zt_leave_start))
+	#time_length_zt_can_still_ping=$((time_zt_can_still_ping-time_zt_leave_start))
+
+	summary=$(
+		cat <<EOF
+{
+  "version":"$ZTO_VER",
+  "commit":"$ZTO_COMMIT",
+  "arch_m":"$(uname -m)",
+  "arch_a":"$(uname -a)",
+  "binary_size":"$(stat -c %s zerotier-one)",
+  "time_length_test":$time_length_test,
+  "time_to_both_nodes_online":$time_to_both_nodes_online,
+  "num_possible_bytes_lost": $POSSIBLY_LOST,
+  "num_definite_bytes_lost": $DEFINITELY_LOST,
+  "num_bad_formattings": $POSSIBLY_LOST,
+  "coverage_lines_covered": $COVERAGE_LINE_COVERED,
+  "coverage_lines_total": $COVERAGE_LINE_TOTAL,
+  "coverage_lines_percent": $COVERAGE_LINE_PERCENT,
+  "ping_loss_percent_1_to_2": $ping_loss_percent_1_to_2,
+  "ping_loss_percent_2_to_1": $ping_loss_percent_2_to_1,
+  "test_exit_code": $1,
+  "test_exit_reason":"$2"
+}
+EOF
+	)
+
+	echo $summary >$FILENAME_SUMMARY
+	cat $FILENAME_SUMMARY
+
+	exit 0
+}
+
+################################################################################
+# CLI Check                                                                    #
+################################################################################
+
+spam_cli() {
+	echo "Spamming CLI..."
+	# Rapidly spam the CLI with joins/leaves
+
+	MAX_TRIES="${1:-10}"
+
+	for ((s = 0; s <= MAX_TRIES; s++)); do
+		$ZT1 status
+		$ZT2 status
+		sleep 0.1
+	done
+
+	SPAM_TRIES=128
+
+	for ((s = 0; s <= SPAM_TRIES; s++)); do
+		$ZT1 join $TEST_NETWORK
+	done
+
+	for ((s = 0; s <= SPAM_TRIES; s++)); do
+		$ZT1 leave $TEST_NETWORK
+	done
+
+	for ((s = 0; s <= SPAM_TRIES; s++)); do
+		$ZT1 leave $TEST_NETWORK
+		$ZT1 join $TEST_NETWORK
+	done
+}
+
+################################################################################
+# Check for proper exit on load of invalid identity                            #
+################################################################################
+
+check_exit_on_invalid_identity() {
+	echo "Checking ZeroTier exits on invalid identity..."
+	mkdir -p $(pwd)/exit_test
+	ZT1="sudo ./zerotier-one -p9999 $(pwd)/exit_test"
+	echo "asdfasdfasdfasdf" > $(pwd)/exit_test/identity.secret
+	echo "asdfasdfasdfasdf" > $(pwd)/exit_test/authtoken.secret
+
+	echo "Launch ZeroTier with an invalid identity"
+	$ZT1 &
+	my_pid=$!
+
+	echo "Waiting 5 seconds"
+	sleep 5
+
+	# check if process is running
+	kill -0 $my_pid
+	if [ $? -eq 0 ]; then
+		exit_test_and_generate_report $TEST_FAIL "Exit test FAILED: Process still running after being fed an invalid identity"
+	fi
+}
+
+test "$@"

.github/workflows/validate-report.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+################################################################################
+# Set exit code depending on tool reports                                      #
+################################################################################
+
+DEFINITELY_LOST=$(cat *test-results/*summary.json | jq .num_definite_bytes_lost)
+EXIT_CODE=$(cat *test-results/*summary.json | jq .exit_code)
+EXIT_REASON=$(cat *test-results/*summary.json | jq .exit_reason)
+
+cat *test-results/*summary.json
+
+echo -e "\nBytes of memory definitely lost: $DEFINITELY_LOST"
+
+if [[ "$DEFINITELY_LOST" -gt 0 ]]; then
+      exit 1
+fi
+
+# Catch-all for other non-zero exit codes
+
+if [[ "$EXIT_CODE" -gt 0 ]]; then
+      echo "Test failed: $EXIT_REASON"
+      exit 1
+fi

.github/workflows/validate.yml

@@ -0,0 +1,56 @@
+on:
+  push:
+  workflow_dispatch:
+
+jobs:
+  build_ubuntu:
+    runs-on: ubuntu-latest
+    steps:
+    - name: gitconfig
+      run: |
+        git config --global core.autocrlf input
+
+    - name: checkout
+      uses: actions/checkout@v3
+      with:
+        fetch-depth: 0
+
+    - name: Install Rust
+      uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+        target: x86_64-unknown-linux-gnu
+        override: true
+        components: rustfmt, clippy
+
+    - name: Set up cargo cache
+      uses: Swatinem/rust-cache@v2
+      continue-on-error: false
+      with:
+        key: ${{ runner.os }}-cargo-${{ hashFiles('zeroidc//Cargo.lock') }}
+        shared-key: ${{ runner.os }}-cargo-
+        workspaces: |
+          zeroidc/
+
+    - name: validate-1m-linux
+      env:
+        CC: 'gcc'
+        CXX: 'g++'
+        BRANCH: ${{  github.ref_name }}
+      run: |
+        sudo apt install -y valgrind xmlstarlet gcovr iperf3 tree
+        make one ZT_COVERAGE=1 ZT_TRACE=1
+        sudo chmod +x ./.github/workflows/validate-linux.sh
+        sudo ./.github/workflows/validate-linux.sh
+
+    - name: Archive test results
+      uses: actions/upload-artifact@v3
+      with:
+        name: ${{github.sha}}-test-results
+        path: "*test-results*"
+
+    - name: final-report
+      run: |
+        sudo chmod +x ./.github/workflows/validate-report.sh
+        sudo ./.github/workflows/validate-report.sh
+

.gitignore

@@ -136,4 +136,5 @@ zeroidc/target/
 __pycache__
 *.pyc
 *_source.tar.bz2
-snap/.snapcraft
+snap/.snapcraft
+tcp-proxy/tcp-proxy

.kick

@@ -0,0 +1,14 @@
+kick
+kick
+kick
+kick
+kick
+kick
+kick
+kick
+kick
+kick
+kick
+kick
+kick
+kick

Makefile

@@ -30,3 +30,4 @@ endif
 drone:
 	@echo "rendering .drone.yaml from .drone.jsonnet"
 	drone jsonnet --format --stream
+	drone sign zerotier/ZeroTierOne --save

README.docker.md

@@ -60,6 +60,7 @@ To ensure you have a network available before trying to listen on it. Without pr
 
 You can control a few settings including the identity used and the authtoken used to interact with the control socket (which you can forward and access through `localhost:9993`).
 
+- `ZEROTIER_JOIN_NETWORKS`: additional way to set networks to join.
 - `ZEROTIER_API_SECRET`: replaces the `authtoken.secret` before booting and allows you to manage the control socket's authentication key.
 - `ZEROTIER_IDENTITY_PUBLIC`: the `identity.public` file for zerotier-one. Use `zerotier-idtool` to generate one of these for you.
 - `ZEROTIER_IDENTITY_SECRET`: the `identity.secret` file for zerotier-one. Use `zerotier-idtool` to generate one of these for you.

README.md

@@ -37,7 +37,6 @@ The base path contains the ZeroTier One service main entry point (`one.cpp`), se
  - `ext/`: third party libraries, binaries that we ship for convenience on some platforms (Mac and Windows), and installation support files.
  - `include/`: include files for the ZeroTier core.
  - `java/`: a JNI wrapper used with our Android mobile app. (The whole Android app is not open source but may be made so in the future.)
- - `macui/`: a Macintosh menu-bar app for controlling ZeroTier One, written in Objective C.
  - `node/`: the ZeroTier virtual Ethernet switch core, which is designed to be entirely separate from the rest of the code and able to be built as a stand-alone OS-independent library. Note to developers: do not use C++11 features in here, since we want this to build on old embedded platforms that lack C++11 support. C++11 can be used elsewhere.
  - `osdep/`: code to support and integrate with OSes, including platform-specific stuff only built for certain targets.
  - `rule-compiler/`: JavaScript rules language compiler for defining network-level rules.
@@ -61,6 +60,7 @@ To build on Mac and Linux just type `make`. On FreeBSD and OpenBSD `gmake` (GNU
    - Rust for x86_64 and ARM64 targets *if SSO is enabled in the build*.
  - **FreeBSD**
    - GNU make is required. Type `gmake` to build.
+   - `binutils` is required.  Type `pkg install binutils` to install.
    - Rust for x86_64 and ARM64 targets *if SSO is enabled in the build*.
  - **OpenBSD**
    - There is a limit of four network memberships on OpenBSD as there are only four tap devices (`/dev/tap0` through `/dev/tap3`).
@@ -104,8 +104,68 @@ On CentOS check `/etc/sysconfig/iptables` for IPTables rules. For other distribu
 
 ZeroTier One peers will automatically locate each other and communicate directly over a local wired LAN *if UDP port 9993 inbound is open*. If that port is filtered, they won't be able to see each others' LAN announcement packets. If you're experiencing poor performance between devices on the same physical network, check their firewall settings. Without LAN auto-location peers must attempt "loopback" NAT traversal, which sometimes fails and in any case requires that every packet traverse your external router twice.
 
-Users behind certain types of firewalls and "symmetric" NAT devices may not able able to connect to external peers directly at all. ZeroTier has limited support for port prediction and will *attempt* to traverse symmetric NATs, but this doesn't always work. If P2P connectivity fails you'll be bouncing UDP packets off our relay servers resulting in slower performance. Some NAT router(s) have a configurable NAT mode, and setting this to "full cone" will eliminate this problem. If you do this you may also see a magical improvement for things like VoIP phones, Skype, BitTorrent, WebRTC, certain games, etc., since all of these use NAT traversal techniques similar to ours.
+Users behind certain types of firewalls and "symmetric" NAT devices may not be able to connect to external peers directly at all. ZeroTier has limited support for port prediction and will *attempt* to traverse symmetric NATs, but this doesn't always work. If P2P connectivity fails you'll be bouncing UDP packets off our relay servers resulting in slower performance. Some NAT router(s) have a configurable NAT mode, and setting this to "full cone" will eliminate this problem. If you do this you may also see a magical improvement for things like VoIP phones, Skype, BitTorrent, WebRTC, certain games, etc., since all of these use NAT traversal techniques similar to ours.
 
 If a firewall between you and the Internet blocks ZeroTier's UDP traffic, you will fall back to last-resort TCP tunneling to rootservers over port 443 (https impersonation). This will work almost anywhere but is *very slow* compared to UDP or direct peer to peer connectivity.
 
 Additional help can be found in our [knowledge base](https://zerotier.atlassian.net/wiki/spaces/SD/overview).
+
+### Prometheus Metrics
+
+Prometheus Metrics are available at the `/metrics` API endpoint.  This endpoint is protected by an API key stored in `metricstoken.secret` to prevent unwanted information leakage.  Information that could be gleaned from the metrics include joined networks and peers your instance is talking to. 
+
+Access control is via the ZeroTier control interface itself and `metricstoken.secret`. This can be sent as a bearer auth token, via the `X-ZT1-Auth` HTTP header field, or appended to the URL as `?auth=<token>`. You can see the current metrics via `cURL` with the following command:
+
+    // Linux
+    curl -H "X-ZT1-Auth: $(sudo cat /var/lib/zerotier-one/metricstoken.secret)" http://localhost:9993/metrics
+
+    // macOS
+    curl -H "X-XT1-Auth: $(sudo cat /Library/Application\ Support/ZeroTier/One/metricstoken.secret)" http://localhost:9993/metrics
+
+    // Windows PowerShell (Admin)
+    Invoke-RestMethod -Headers @{'X-ZT1-Auth' = "$(Get-Content C:\ProgramData\ZeroTier\One\metricstoken.secret)"; } -Uri http://localhost:9993/metrics
+
+To configure a scrape job in Prometheus on the machine ZeroTier is running on, add this to your Prometheus `scrape_config`:
+
+    - job_name: zerotier-one
+      honor_labels: true
+      scrape_interval: 15s
+      metrics_path: /metrics
+      static_configs:
+      - targets:
+        - 127.0.0.1:9993
+        labels:
+          group: zerotier-one
+          node_id: $YOUR_10_CHARACTER_NODE_ID
+      authorization:
+        credentials: $YOUR_METRICS_TOKEN_SECRET
+
+If neither of these methods are desirable, it is probably possible to distribute metrics via [Prometheus Proxy](https://github.com/pambrose/prometheus-proxy) or some other tool.  Note: We have not tested this internally, but will probably work with the correct configuration.
+
+Metrics are also available on disk in ZeroTier's working directory:
+
+   // Linux
+   /var/lib/zerotier-one/metrics.prom
+
+   // macOS
+   /Library/Application\ Support/ZeroTier/One/metrics.prom
+
+   //Windows
+   C:\ProgramData\ZeroTier\One\metrics.prom
+
+#### Available Metrics
+
+| Metric Name | Labels | Metric Type | Description |
+| ---         | ---    | ---         | ---         |
+| zt_packet | packet_type, direction | Counter | ZeroTier packet type counts |
+| zt_packet_error | error_type, direction | Counter | ZeroTier packet errors|
+| zt_data | protocol, direction | Counter | number of bytes ZeroTier has transmitted or received |
+| zt_num_networks | | Gauge | number of networks this instance is joined to |
+| zt_network_multicast_groups_subscribed | network_id | Gauge | number of multicast groups networks are subscribed to |
+| zt_network_packets | network_id, direction | Counter | number of incoming/outgoing packets per network |
+| zt_peer_latency | node_id | Histogram | peer latency (ms) |
+| zt_peer_path_count | node_id, status | Gauge | number of paths to peer |
+| zt_peer_packets | node_id, direction | Counter | number of packets to/from a peer |
+| zt_peer_packet_errors | node_id | Counter | number of incoming packet errors from a peer |
+
+If there are other metrics you'd like to see tracked, ask us in an Issue or send us a Pull Request!

RELEASE-NOTES.md

@@ -1,6 +1,51 @@
 ZeroTier Release Notes
 ======
 
+# 2023-08-23 -- Version 1.12.0
+
+  * Experimental Windows ARM64 support
+  * Fix numerous sleep/wake issues on macOS and other platforms
+  * Faster recovery after changes to physical network settings
+  * Prometheus compatible metrics support!
+  * Fix full tunnel mode on recent macOS versions
+  * Numerous macOS DNS fixes
+  * 10-30% speed improvement on Linux
+
+# 2023-03-23 -- Version 1.10.6
+
+  * Prevent binding temporary ipv6 addresses on macos (#1910)
+  * Prevent path-learning loops (#1914)
+  * Prevent infinite loop of UAC prompts in tray app
+
+# 2023-03-10 -- Version 1.10.5
+
+ * Fix for high CPU usage bug on Windows
+
+# 2023-03-07 -- Version 1.10.4
+
+ * SECURITY FIX (Windows): this version fixes a file permission problem on
+   Windows that could allow non-privileged users on a Windows system to read
+   privileged files in the ZeroTier service's working directory. This could
+   allow an unprivileged local Windows user to administrate the local ZeroTier
+   instance without appropriate local permissions. This issue is not remotely
+   exploitable unless a remote user can read arbitrary local files, and does
+   not impact other operating systems.
+
+ * Fix a bug in the handling of multiple IP address assignments to virtual
+   interfaces on macOS.
+
+# 2023-02-15 -- Version 1.10.3
+
+ * Fix for duplicate paths in client. Could cause connectivity issues. Affects all platforms.
+ * Fix for Ethernet Tap MTU setting, would not properly apply on Linux.
+ * Fix default route bugs (macOS.)
+ * Enable Ping automatically for ZeroTier Adapters (Windows.)
+ * SSO updates and minor bugfixes.
+ * Add low-bandwidth mode.
+ * Add forceTcpRelay mode (optionally enabled.)
+ * Fix bug that prevented setting of custom TCP relay address.
+ * Build script improvements and bug fixes.
+
 # 2022-11-01 -- Version 1.10.2
 
  * Fix another SSO "stuck client" issue in zeroidc.
@@ -32,7 +77,7 @@ Note that releases are coming few and far between because most of our dev effort
 
 # 2022-04-25 -- Version 1.8.9
 
- * Fixed a long-standing and strange bug that was causing sporadic "phantom" packet authentication failures. Not a security problem but could be behind spordaic reports of link failures under some conditions.
+ * Fixed a long-standing and strange bug that was causing sporadic "phantom" packet authentication failures. Not a security problem but could be behind sporadic reports of link failures under some conditions.
  * Fized a memory leak in SSO/OIDC support.
  * Fixed SSO/OIDC display error on CLI.
  * Fixed a bug causing nodes to sometimes fail to push certs to each other (primarily affects SSO/OIDC use cases).

SECURITY.md

@@ -0,0 +1,95 @@
+# Security
+
+ZeroTier takes the security of our software products and services seriously, which 
+includes all source code repositories managed through our GitHub organization.
+
+## Supported Versions
+
+The following versions of ZeroTier One receive security updates
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 1.10.x   | :white_check_mark: |
+| 1.8.x   | :white_check_mark:  |
+| < 1.8.0 | :x: |
+
+## Reporting a Vulnerability
+
+**Please do not report security issues through public GitHub issues**
+
+Instead, please report vulnerabilities via email to security@zerotier.com. If possible,
+please encrypt with our PGP key (see below).
+
+Please include the following information, or as much as you can provide to help us 
+understand the nature and scope of the issue:
+
+
+  * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
+  * Full paths of source file(s) related to the manifestation of the issue
+  * The location of the affected source code (tag/branch/commit or direct URL)
+  * Any special configuration required to reproduce the issue
+  * Step-by-step instructions to reproduce the issue
+  * Proof-of-concept or exploit code (if possible)
+  * Impact of the issue, including how an attacker might exploit the issue
+
+
+## Preferred Languages
+
+We prefer all communications to be in English.
+
+## security@zerotier.com PGP key
+
+```
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mQINBGQGOVIBEACalXTnNqaiSOVLFEiqHpDMg8N/OI5D5850Xy1ZEvx3B3rz7cbn
+k30ozHtJKbh+vqpyItE7DjyQAuF19gP5Q64Yh0Y+MmLHq60q/GwOwAYz7cI+UzA3
+5x8YqcmTp32LAM1xJn+iMlMLBuAmJl4kULKmOXPlpqPiyTFs5saizvm7fgRmfgJJ
+HpsnIrTkaDFJhAR+jvMJohVYwmhuydeI0DsHu7KGpG1ddcHDrUjOPNqXnnAPSPwx
+llw4yfKlQb8GYErsv/G5QVyzd5+SxEuiI4MARRnrk8LlMQ33CR6pzIQ/Bk5AAmye
+mHqfEAknkiOf++urYhRs9BL3Kz3MdV0cg92zr9EFOg0u56jxf5OnAiTOhGUUA0hn
+dS7peVGl46R9Oy2JYIazNDGi+4NIsYDFXsnsss9xOQVygPyeQd71zFHfix0jct9w
+j3o/kj7Egsnm9nc13354bYT6bbalqXiRWwGH1eAFpjueNWiVFwZS6NZUP3WeNDiY
+BlPo1LodvolbXiJcTILTCyEkERJPCK2zoE2nTdVfvTLWsuehw1M6Yd2/q74TVYy/
+RY+KjHkrChEBQ9PqXsXRHj6opKbT8JLfZkvU5k+3IiqqxOpB+QXFI/whj493CxWW
+so7QAmzOCyJq8GDVPxzkwUac22YIkXdiOmb8i/HWq+kLY/HjQE259Gx6KwARAQAB
+tClaZXJvVGllciBTZWN1cml0eSA8c2VjdXJpdHlAemVyb3RpZXIuY29tPokCTAQT
+AQoANhYhBH1HQGb+4jzl6mnFqf09m6uqADkABQJkBjlSAhsDBAsJCAcEFQoJCAUW
+AgMBAAIeAQIXgAAKCRD9PZurqgA5ACqPD/sFt6SG6Tu0HwTY2ofJtYsa2GBLL0pf
+dYlX4cWSs1PVB5+m5Oj18y+GB2umA9GnsVtmvaSfp3XEngt2zNWX27uUsVfL35b2
+/5TVVe8RjzOedqMN+lQWMvO+f/C1zmWYXjjpC+iGjgMMaRRrofkkn+7uL4N9y6gY
+rcXtpACT1rYFC+i1AKnZfUO8Vr5ji7odq0f7bDkN/N38rB0kRRwEmO8wqdpQK6gK
+nxf9vgJl5ggimDk5Xtz1sfd3y28bf5N4hdOCkXUbd10nUFY3wDNTM4VxozxTGJeG
+imdcc19Wuw/1fGUZ5SIjgPanCdPLGYwSTr+M6Fuern9uTtlC1GOby3BUtmVGP6EU
+1pSAJSRpmoBPHKKOYtSMwV8PCboXru9P1ab8y8STKM3SKyghUJrl17gdc0LaksZa
+E54pJudGPIQMFRqZjMdV6jgMuaLTozjZ4mW8EThf4mkX4xDkO8l7cOn0225ZYJZC
+lZKpdnwzk9owkJA80u4KBNJxTtB4ZAPzjBsD5hFzCZQTLNQp/psU3EjZsau28eXT
+E/C1QjEQHgy4ohkgQlCm1H1+clKssCWcdmsVGXuS1u8gh4K6X9b0Z6LeCGRaQvH2
++DB8oTAdqp9nUZv9rP4pbo+sR4fF67CFLriVuxjedAiFkbM4uHMFcL4tc/X9+DRo
+YN5X7oEkZvO507kCDQRkBjlSARAAz58UMF7K1qKyQjzKTcutaYZ5SaIGky9lCLZn
+/2vjpFCoBogkxS/6IKQcwZk8b4S9QstaaQZDFEkxqNeKC0GiFTAMAb6SmYcK495h
+EZnHl0NA5Nc2dBlZk5E/ENzTCz2bXaxCcVESc2z+xCzu07brbhGrqvliKiwOUzt9
+JzqEsar6I95OutBcZvkFCs44/Uf9bS1qf1w4klE8w3vdMtGH23umrET4tFZ+sh6o
+ZFtQx0u2eKjsRdn/RMtsxLNaJlcE1DdIAqBpQrcmuwMC8v5wUGfCGZjhClzmyQlq
+akUkayir7UtbHbFT/mgO+YI77YGXWk5QrwPscqqT2l8KB/YMujNDmaWa/0KV1lIY
+zr5s4dzVeiwqFLR9ANFIhzFwzf3JLi6XSx123Qix0TxZoYPZCHl7yoi9qi6qybz5
+0Od2LSz3jbApeKYymZ+zjE+YV5y9DI6Wzy1j2M1FogNvTO9fMk+6dLt4HhTdSNvH
+cKya462YCcy+tnZTkhmh+FTebbJlV6D4wG7skE5KCdBhjm53xLwp6XW9L6n2CrkL
+W1IDBcCz0oPd1sMkXbO3wnxdXprV2XurCfsg/R2nszSNzvdJ8/xj3cr9hpoJ714R
+qqyoEDRZ1Ss9kGL166o5MpN5qb/EewdkqGgWP7YFXbhsdHQiW7Z7dAqzjoaybD4O
+nakkwyUAEQEAAYkCNgQYAQoAIBYhBH1HQGb+4jzl6mnFqf09m6uqADkABQJkBjlS
+AhsMAAoJEP09m6uqADkAax0P/Rh8EZYRqW6dPYTl1YQusAK10rAcRNq3ekjofXGk
+oXK1S7HWGoFgl5++5nfSfNgFJ5VLcgIM56wtIf49zFjWe5oC6fw8k+ghh4d2chMP
+hdDILx6e0c30Iq1+EvovGR9hWa0wJ4cKTdzlwhY9ZC09q0ia+bl2mwpie1JQDR0c
+zXCjt+PldLeeK9z1/XT0Q7KowYC+U18oR+KFm+EaRV4QT85JVequnIeGkmaHJrHB
+lH4T5A5ib7y8edon1c0Zx3GsaxJUojkEJ0SX7ffVDu6ztUZfkHfCVpMW4VzUeGA/
+m+CtFO9ciLRGZEkRa+zhIGoBvwEXU0GiwiF4nZ0F2C8UioeW0YIEV9zl3nXJctYE
+ZKc2whSENQRTGgaYHVoVZhznt71LKWgFLshwBo81UCXVkzwAjMW1ActDnmPw5M7q
+xR5Qp5G49Z1GmfSozazha0HVFPKNV5i3RlTzs4yLUnZyH0yC9IvtOefMHcLjG96L
+N5miEV97gvJJjrn8rhRvpUwAWgmT/9IuYjBNQTtNN40arto5HxezR76WCjdKYxdL
+p3dM1iiBDShHNm7LdyZlLFhTOMU0tNBxJJ7B09ar5gakeZjD+2aB1ODX9VuFtozL
+onBjI2gIkry0UIkuznHfFw05lZAZAiqHEVgVi/WTk4C/bklDZNgE0lx+IWzEz2iS
+L455
+=lheL
+-----END PGP PUBLIC KEY BLOCK-----
+```

attic/world/build.sh

@@ -1,3 +1,3 @@
 #!/bin/bash
 
-c++ -std=c++11 -I../.. -I.. -g -o mkworld ../../node/C25519.cpp ../../node/Salsa20.cpp ../../node/SHA512.cpp ../../node/Identity.cpp ../../node/Utils.cpp ../../node/InetAddress.cpp ../../osdep/OSUtils.cpp mkworld.cpp -lm
+c++ -std=c++11 -I../.. -I../../ext -I.. -g -o mkworld ../../node/C25519.cpp ../../node/Salsa20.cpp ../../node/SHA512.cpp ../../node/Identity.cpp ../../node/Utils.cpp ../../node/InetAddress.cpp ../../osdep/OSUtils.cpp mkworld.cpp -lm

ci/Dockerfile-build.deb

@@ -0,0 +1,13 @@
+ARG ZT_NAME
+FROM 084037375216.dkr.ecr.us-east-2.amazonaws.com/${ZT_NAME}-builder as builder
+WORKDIR /work/build
+COPY . .
+RUN pwd
+RUN ls -la .
+RUN make clean
+RUN make debian
+RUN ls -ls /work
+
+FROM scratch AS export
+ARG ZT_NAME
+COPY --from=builder /work/*.deb ./${ZT_NAME}/

ci/Dockerfile-build.el6

@@ -0,0 +1,36 @@
+ARG DOCKER_ARCH
+FROM --platform=linux/${DOCKER_ARCH} alpine:edge AS builder
+
+RUN apk update
+RUN apk add curl
+RUN apk add bash
+RUN apk add file
+RUN apk add rust
+RUN apk add cargo
+RUN apk add make
+RUN apk add cmake
+RUN apk add clang
+RUN apk add openssl-dev
+RUN apk add linux-headers
+RUN apk add build-base
+RUN apk add openssl-libs-static
+
+COPY . .
+RUN ZT_STATIC=1 make
+RUN ls -la
+
+ARG DOCKER_ARCH
+FROM --platform=linux/${DOCKER_ARCH} centos:6 AS stage
+WORKDIR /root/rpmbuild/BUILD
+COPY . .
+COPY --from=builder zerotier-* ./
+RUN curl https://gist.githubusercontent.com/someara/b363002ba6e57b3c474dd027d4daef85/raw/4ac5534139752fc92fbe1a53599a390214f69615/el6%2520vault --output /etc/yum.repos.d/CentOS-Base.repo
+RUN uname -a
+RUN yum -y install make gcc rpm-build
+RUN pwd
+RUN ls -la
+RUN make redhat
+
+FROM scratch AS export
+ARG ZT_NAME
+COPY --from=stage /root/rpmbuild/RPMS/*/*.rpm ./${ZT_NAME}/

ci/Dockerfile-build.rpm

@@ -0,0 +1,9 @@
+ARG ZT_NAME
+FROM 084037375216.dkr.ecr.us-east-2.amazonaws.com/${ZT_NAME}-builder as builder
+WORKDIR /root/rpmbuild/BUILD
+COPY . .
+RUN make redhat
+
+FROM scratch AS export
+ARG ZT_NAME
+COPY --from=builder /root/rpmbuild/RPMS/*/*.rpm ./${ZT_NAME}/

ci/Dockerfile-test.deb

@@ -0,0 +1,13 @@
+ARG ZT_NAME
+FROM 084037375216.dkr.ecr.us-east-2.amazonaws.com/${ZT_NAME}-tester
+ARG BASEURL
+ARG VERSION
+ARG DEB_ARCH
+ARG ZT_NAME
+ARG DISTRO
+RUN curl -s http://${BASEURL}/key.gpg -o /etc/apt/trusted.gpg.d/zerotier.gpg
+RUN echo "deb [arch=${DEB_ARCH} signed-by=/etc/apt/trusted.gpg.d/zerotier.gpg] http://${BASEURL}/${DISTRO} ${ZT_NAME} main" > /etc/apt/sources.list.d/zerotier.list
+RUN apt-get -qq update
+RUN apt-get -qq install zerotier-one=${VERSION}
+
+RUN ldd $(which zerotier-cli)

ci/Dockerfile-test.el6

@@ -0,0 +1,4 @@
+ARG DOCKER_ARCH
+FROM --platform=linux/${DOCKER_ARCH} centos:6
+RUN printf "[C6.10-base]\nname=CentOS-6.10 - Base\nbaseurl=http://vault.epel.cloud/6.10/os/\$basearch/\ngpgcheck=1\ngpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6\nenabled=1\nmetadata_expire=never\n" > /etc/yum.repos.d/CentOS-Base.repo
+RUN yum -y install curl

ci/Dockerfile-test.rpm

@@ -0,0 +1,17 @@
+ARG ZT_NAME
+FROM 084037375216.dkr.ecr.us-east-2.amazonaws.com/${ZT_NAME}-tester
+ARG BASEURL
+ARG VERSION
+ARG DEB_ARCH
+ARG ZT_NAME
+ARG DISTRO
+ARG DNF_ARCH
+RUN curl -s http://${BASEURL}/key.asc -o /etc/pki/rpm-gpg/RPM-GPG-KEY-zerotier
+RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-zerotier
+RUN rpm -q gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n'
+RUN printf "[zerotier]\nname=zerotier\nbaseurl=http://${BASEURL}/${DISTRO}/${ZT_NAME}/$basearch/\nenabled=1\ngpgcheck=1\ngpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zerotier\n" > /etc/yum.repos.d/zerotier.repo
+
+# RUN yum -v repolist
+RUN setarch ${DNF_ARCH} yum -y install zerotier-one-${VERSION}
+RUN file $(which zerotier-cli)
+RUN ldd $(which zerotier-cli)