VERSION 1.2.4: Bug fixes and performance improvements

* Managed routes are now only bifurcated for the default route. This is a change in behavior, though few people will probably notice. Bifurcating all managed routes was causing more trouble than it was worth for most users.
 * Up to 2X crypto speedup on x86-64 (except Windows, which will take some porting) and 32-bit ARM platforms due to integration of fast assembly language implementations of Salsa20/12 from the [supercop](http://bench.cr.yp.to/supercop.html) code base. These were written by Daniel J. Bernstein and are in the public domain. My Macbook Pro (Core i5 2.8ghz) now does almost 1.5GiB/sec Salsa20/12 per core and a Raspberry Pi got a 2X boost. 64-bit ARM support and Windows support will take some work but should not be too hard.
 * Refactored code that manages credentials to greatly reduce memory use in most cases. This may also result in a small performance improvement.
 * Reworked and simplified path selection and priority logic to fix path instability and dead path persistence edge cases. There have been some sporadic reports of persistent path instabilities and dead paths hanging around that take minutes to resolve. These have proven difficult to reproduce in house, but hopefully this will fix them. In any case it seems to speed up path establishment in our tests and it makes the code simpler and more readable.
 * Eliminated some unused cruft from the code around path management and in the peer class.
 * Fixed an issue causing build problems on some MIPS architecture systems.
 * Fixed Windows forgetting routes on sleep/wake or in some other circumstances. (GitHub issue #465)

.gitignore

@@ -60,7 +60,7 @@ zt1-src.tar.gz
 *.pid
 *.pkg
 *.o
-*.a
+/*.a
 *.dylib
 *.so
 *.so.*
@@ -78,6 +78,7 @@ debian/zerotier-one
 debian/zerotier-one*.debhelper
 debian/*.log
 debian/zerotier-one.substvars
+root-watcher/config.json
 
 # Java/Android/JNI build droppings
 java/obj/

RELEASE-NOTES.md

@@ -1,16 +1,22 @@
 ZeroTier Release Notes
 ======
 
-# 2017-03-17 -- Version 1.2.2
+# 2017-04-20 -- Version 1.2.4
 
-Version 1.2.2 fixes a few bugs discovered after the 1.2.0 release. These are:
+ * Managed routes are now only bifurcated for the default route. This is a change in behavior, though few people will probably notice. Bifurcating all managed routes was causing more trouble than it was worth for most users.
+ * Up to 2X crypto speedup on x86-64 (except Windows, which will take some porting) and 32-bit ARM platforms due to integration of fast assembly language implementations of Salsa20/12 from the [supercop](http://bench.cr.yp.to/supercop.html) code base. These were written by Daniel J. Bernstein and are in the public domain. My Macbook Pro (Core i5 2.8ghz) now does almost 1.5GiB/sec Salsa20/12 per core and a Raspberry Pi got a 2X boost. 64-bit ARM support and Windows support will take some work but should not be too hard.
+ * Refactored code that manages credentials to greatly reduce memory use in most cases. This may also result in a small performance improvement.
+ * Reworked and simplified path selection and priority logic to fix path instability and dead path persistence edge cases. There have been some sporadic reports of persistent path instabilities and dead paths hanging around that take minutes to resolve. These have proven difficult to reproduce in house, but hopefully this will fix them. In any case it seems to speed up path establishment in our tests and it makes the code simpler and more readable.
+ * Eliminated some unused cruft from the code around path management and in the peer class.
+ * Fixed an issue causing build problems on some MIPS architecture systems.
+ * Fixed Windows forgetting routes on sleep/wake or in some other circumstances. (GitHub issue #465)
+
+# 2017-03-17 -- Version 1.2.2
 
  * A bug causing unreliable multicast propagation (GitHub issue #461).
  * A crash in ARM binaries due to a build chain and flags problem.
  * A bug in the network controller preventing members from being listed (GitHub issue #460).
 
-------
-
 # 2017-03-14 -- Version 1.2.0
 
 Version 1.2.0 is a major milestone release representing almost nine months of work. It includes our rules engine for distributed network packet filtering and security monitoring, federated roots, and many other architectural and UI improvements and bug fixes.

controller/EmbeddedNetworkController.cpp

@@ -61,9 +61,6 @@ using json = nlohmann::json;
 // Nodes are considered active if they've queried in less than this long
 #define ZT_NETCONF_NODE_ACTIVE_THRESHOLD (ZT_NETWORK_AUTOCONF_DELAY * 2)
 
-// Timeout for disk read cache (ms)
-#define ZT_NETCONF_DB_CACHE_TTL 60000
-
 namespace ZeroTier {
 
 static json _renderRule(ZT_VirtualNetworkRule &rule)
@@ -432,12 +429,11 @@ static bool _parseRule(json &r,ZT_VirtualNetworkRule &rule)
 }
 
 EmbeddedNetworkController::EmbeddedNetworkController(Node *node,const char *dbPath) :
+	_startTime(OSUtils::now()),
 	_threadsStarted(false),
 	_db(dbPath),
 	_node(node)
 {
-	OSUtils::mkdir(dbPath);
-	OSUtils::lockDownFile(dbPath,true); // networks might contain auth tokens, etc., so restrict directory permissions
 }
 
 EmbeddedNetworkController::~EmbeddedNetworkController()
@@ -503,7 +499,7 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpGET(
 			json network;
 			{
 				Mutex::Lock _l(_db_m);
-				network = _db.get("network",nwids,ZT_NETCONF_DB_CACHE_TTL);
+				network = _db.get("network",nwids);
 			}
 			if (!network.size())
 				return 404;
@@ -518,7 +514,7 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpGET(
 						json member;
 						{
 							Mutex::Lock _l(_db_m);
-							member = _db.get("network",nwids,"member",Address(address).toString(),ZT_NETCONF_DB_CACHE_TTL);
+							member = _db.get("network",nwids,"member",Address(address).toString());
 						}
 						if (!member.size())
 							return 404;
@@ -533,7 +529,7 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpGET(
 						Mutex::Lock _l(_db_m);
 
 						responseBody = "{";
-						_db.filter((std::string("network/") + nwids + "/member/"),ZT_NETCONF_DB_CACHE_TTL,[&responseBody](const std::string &n,const json &member) {
+						_db.filter((std::string("network/") + nwids + "/member/"),[&responseBody](const std::string &n,const json &member) {
 							if ((member.is_object())&&(member.size() > 0)) {
 								responseBody.append((responseBody.length() == 1) ? "\"" : ",\"");
 								responseBody.append(OSUtils::jsonString(member["id"],"0"));
@@ -566,7 +562,7 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpGET(
 			std::set<std::string> networkIds;
 			{
 				Mutex::Lock _l(_db_m);
-				_db.filter("network/",120000,[&networkIds](const std::string &n,const json &obj) {
+				_db.filter("network/",[&networkIds](const std::string &n,const json &obj) {
 					if (n.length() == (16 + 8))
 						networkIds.insert(n.substr(8));
 					return true; // do not delete
@@ -641,7 +637,7 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpPOST(
 					json member;
 					{
 						Mutex::Lock _l(_db_m);
-						member = _db.get("network",nwids,"member",Address(address).toString(),ZT_NETCONF_DB_CACHE_TTL);
+						member = _db.get("network",nwids,"member",Address(address).toString());
 					}
 					json origMember(member); // for detecting changes
 					_initMember(member);
@@ -667,7 +663,7 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpPOST(
 								// Member is being de-authorized, so spray Revocation objects to all online members
 								if (!newAuth) {
 									_clearNetworkMemberInfoCache(nwid);
-									Revocation rev(_node->prng(),nwid,0,now,ZT_REVOCATION_FLAG_FAST_PROPAGATE,Address(address),Revocation::CREDENTIAL_TYPE_COM);
+									Revocation rev((uint32_t)_node->prng(),nwid,0,now,ZT_REVOCATION_FLAG_FAST_PROPAGATE,Address(address),Revocation::CREDENTIAL_TYPE_COM);
 									rev.sign(_signingId);
 									Mutex::Lock _l(_lastRequestTime_m);
 									for(std::map< std::pair<uint64_t,uint64_t>,uint64_t >::iterator i(_lastRequestTime.begin());i!=_lastRequestTime.end();++i) {
@@ -792,7 +788,7 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpPOST(
 					test->timestamp = OSUtils::now();
 
 					if (_node) {
-						_node->circuitTestBegin(test,&(EmbeddedNetworkController::_circuitTestCallback));
+						_node->circuitTestBegin((void *)0,test,&(EmbeddedNetworkController::_circuitTestCallback));
 					} else {
 						_tests.pop_back();
 						return 500;
@@ -824,7 +820,7 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpPOST(
 							uint64_t tryNwid = nwidPrefix | (nwidPostfix & 0xffffffULL);
 							if ((tryNwid & 0xffffffULL) == 0ULL) tryNwid |= 1ULL;
 							Utils::snprintf(nwids,sizeof(nwids),"%.16llx",(unsigned long long)tryNwid);
-							if (_db.get("network",nwids,ZT_NETCONF_DB_CACHE_TTL).size() <= 0) {
+							if (_db.get("network",nwids).size() <= 0) {
 								nwid = tryNwid;
 								break;
 							}
@@ -833,7 +829,7 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpPOST(
 							return 503;
 					}
 
-					network = _db.get("network",nwids,ZT_NETCONF_DB_CACHE_TTL);
+					network = _db.get("network",nwids);
 				}
 				json origNetwork(network); // for detecting changes
 				_initNetwork(network);
@@ -1020,9 +1016,10 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpPOST(
 									json ntag = json::object();
 									const uint64_t tagId = OSUtils::jsonInt(tag["id"],0ULL);
 									ntag["id"] = tagId;
-									if (tag.find("default") == tag.end())
-										ntag["default"] = json();
-									else ntag["default"] = OSUtils::jsonInt(tag["default"],0ULL);
+									json &dfl = tag["default"];
+									if (dfl.is_null())
+										ntag["default"] = dfl;
+									else ntag["default"] = OSUtils::jsonInt(dfl,0ULL);
 									ntags[tagId] = ntag;
 								}
 							}
@@ -1053,7 +1050,7 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpPOST(
 					}
 
 					// Send an update to all members of the network
-					_db.filter((std::string("network/") + nwids + "/member/"),120000,[this,&now,&nwid](const std::string &n,const json &obj) {
+					_db.filter((std::string("network/") + nwids + "/member/"),[this,&now,&nwid](const std::string &n,const json &obj) {
 						_pushMemberUpdate(now,nwid,obj);
 						return true; // do not delete
 					});
@@ -1070,7 +1067,19 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpPOST(
 
 		} // else 404
 
-	} // else 404
+	} else if (path[0] == "ping") {
+
+		json testRec;
+		const uint64_t now = OSUtils::now();
+		testRec["clock"] = now;
+		testRec["uptime"] = (now - _startTime);
+		testRec["content"] = b;
+		responseBody = OSUtils::jsonDump(testRec);
+		_db.writeRaw("pong",responseBody);
+		responseContentType = "application/json";
+		return 200;
+
+	}
 
 	return 404;
 }
@@ -1095,7 +1104,7 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpDELETE(
 			json network;
 			{
 				Mutex::Lock _l(_db_m);
-				network = _db.get("network",nwids,ZT_NETCONF_DB_CACHE_TTL);
+				network = _db.get("network",nwids);
 			}
 			if (!network.size())
 				return 404;
@@ -1106,7 +1115,7 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpDELETE(
 
 					Mutex::Lock _l(_db_m);
 
-					json member = _db.get("network",nwids,"member",Address(address).toString(),ZT_NETCONF_DB_CACHE_TTL);
+					json member = _db.get("network",nwids,"member",Address(address).toString());
 					_db.erase("network",nwids,"member",Address(address).toString());
 
 					if (!member.size())
@@ -1118,8 +1127,9 @@ unsigned int EmbeddedNetworkController::handleControlPlaneHttpDELETE(
 			} else {
 				Mutex::Lock _l(_db_m);
 
-				std::string pfx("network/"); pfx.append(nwids);
-				_db.filter(pfx,120000,[](const std::string &n,const json &obj) {
+				std::string pfx("network/");
+				pfx.append(nwids);
+				_db.filter(pfx,[](const std::string &n,const json &obj) {
 					return false; // delete
 				});
 
@@ -1246,8 +1256,8 @@ void EmbeddedNetworkController::_request(
 	json member;
 	{
 		Mutex::Lock _l(_db_m);
-		network = _db.get("network",nwids,ZT_NETCONF_DB_CACHE_TTL);
-		member = _db.get("network",nwids,"member",identity.address().toString(),ZT_NETCONF_DB_CACHE_TTL);
+		network = _db.get("network",nwids);
+		member = _db.get("network",nwids,"member",identity.address().toString());
 	}
 
 	if (!network.size()) {
@@ -1772,7 +1782,7 @@ void EmbeddedNetworkController::_getNetworkMemberInfo(uint64_t now,uint64_t nwid
 
 	{
 		Mutex::Lock _l(_db_m);
-		_db.filter(pfx,120000,[&nmi,&now](const std::string &n,const json &member) {
+		_db.filter(pfx,[&nmi,&now](const std::string &n,const json &member) {
 			try {
 				if (OSUtils::jsonBool(member["authorized"],false)) {
 					++nmi.authorizedMemberCount;

controller/EmbeddedNetworkController.hpp

@@ -98,14 +98,6 @@ public:
 		throw();
 
 private:
-	static void _circuitTestCallback(ZT_Node *node,ZT_CircuitTest *test,const ZT_CircuitTestReport *report);
-	void _request(
-		uint64_t nwid,
-		const InetAddress &fromAddr,
-		uint64_t requestPacketId,
-		const Identity &identity,
-		const Dictionary<ZT_NETWORKCONFIG_METADATA_DICT_CAPACITY> &metaData);
-
 	struct _RQEntry
 	{
 		uint64_t nwid;
@@ -114,11 +106,6 @@ private:
 		Identity identity;
 		Dictionary<ZT_NETWORKCONFIG_METADATA_DICT_CAPACITY> metaData;
 	};
-	BlockingQueue<_RQEntry *> _queue;
-
-	Thread _threads[ZT_EMBEDDEDNETWORKCONTROLLER_BACKGROUND_THREAD_COUNT];
-	bool _threadsStarted;
-	Mutex _threads_m;
 
 	// Gathers a bunch of statistics about members of a network, IP assignments, etc. that we need in various places
 	struct _NetworkMemberInfo
@@ -132,15 +119,11 @@ private:
 		uint64_t mostRecentDeauthTime;
 		uint64_t nmiTimestamp; // time this NMI structure was computed
 	};
-	std::map<uint64_t,_NetworkMemberInfo> _nmiCache;
-	Mutex _nmiCache_m;
-	void _getNetworkMemberInfo(uint64_t now,uint64_t nwid,_NetworkMemberInfo &nmi);
-	inline void _clearNetworkMemberInfoCache(const uint64_t nwid)
-	{
-		Mutex::Lock _l(_nmiCache_m);
-		_nmiCache.erase(nwid);
-	}
 
+	static void _circuitTestCallback(ZT_Node *node,ZT_CircuitTest *test,const ZT_CircuitTestReport *report);
+	void _request(uint64_t nwid,const InetAddress &fromAddr,uint64_t requestPacketId,const Identity &identity,const Dictionary<ZT_NETWORKCONFIG_METADATA_DICT_CAPACITY> &metaData);
+	void _getNetworkMemberInfo(uint64_t now,uint64_t nwid,_NetworkMemberInfo &nmi);
+	inline void _clearNetworkMemberInfoCache(const uint64_t nwid) { Mutex::Lock _l(_nmiCache_m); _nmiCache.erase(nwid); }
 	void _pushMemberUpdate(uint64_t now,uint64_t nwid,const nlohmann::json &member);
 
 	// These init objects with default and static/informational fields
@@ -196,6 +179,16 @@ private:
 		member["clock"] = now;
 	}
 
+	const uint64_t _startTime;
+
+	BlockingQueue<_RQEntry *> _queue;
+	Thread _threads[ZT_EMBEDDEDNETWORKCONTROLLER_BACKGROUND_THREAD_COUNT];
+	bool _threadsStarted;
+	Mutex _threads_m;
+
+	std::map<uint64_t,_NetworkMemberInfo> _nmiCache;
+	Mutex _nmiCache_m;
+
 	JSONDB _db;
 	Mutex _db_m;
 

controller/JSONDB.cpp

@@ -18,98 +18,104 @@
 
 #include "JSONDB.hpp"
 
+#define ZT_JSONDB_HTTP_TIMEOUT 60000
+
 namespace ZeroTier {
 
 static const nlohmann::json _EMPTY_JSON(nlohmann::json::object());
+static const std::map<std::string,std::string> _ZT_JSONDB_GET_HEADERS;
+
+JSONDB::JSONDB(const std::string &basePath) :
+	_basePath(basePath),
+	_ready(false)
+{
+	if ((_basePath.length() > 7)&&(_basePath.substr(0,7) == "http://")) {
+		// TODO: this doesn't yet support IPv6 since bracketed address notiation isn't supported.
+		// Typically it's used with 127.0.0.1 anyway.
+		std::string hn = _basePath.substr(7);
+		std::size_t hnend = hn.find_first_of('/');
+		if (hnend != std::string::npos)
+			hn = hn.substr(0,hnend);
+		std::size_t hnsep = hn.find_last_of(':');
+		if (hnsep != std::string::npos)
+			hn[hnsep] = '/';
+		_httpAddr.fromString(hn);
+		if (hnend != std::string::npos)
+			_basePath = _basePath.substr(7 + hnend);
+		if (_basePath.length() == 0)
+			_basePath = "/";
+		if (_basePath[0] != '/')
+			_basePath = std::string("/") + _basePath;
+	} else {
+		OSUtils::mkdir(_basePath.c_str());
+		OSUtils::lockDownFile(_basePath.c_str(),true); // networks might contain auth tokens, etc., so restrict directory permissions
+	}
+	_ready = _reload(_basePath,std::string());
+}
 
 bool JSONDB::writeRaw(const std::string &n,const std::string &obj)
 {
 	if (!_isValidObjectName(n))
 		return false;
-
-	const std::string path(_genPath(n,true));
-	if (!path.length())
-		return false;
-
-	const std::string buf(obj);
-	if (!OSUtils::writeFile(path.c_str(),buf))
-		return false;
-
-	return true;
+	if (_httpAddr) {
+		std::map<std::string,std::string> headers;
+		std::string body;
+		std::map<std::string,std::string> reqHeaders;
+		char tmp[64];
+		Utils::snprintf(tmp,sizeof(tmp),"%lu",(unsigned long)obj.length());
+		reqHeaders["Content-Length"] = tmp;
+		reqHeaders["Content-Type"] = "application/json";
+		const unsigned int sc = Http::PUT(1048576,ZT_JSONDB_HTTP_TIMEOUT,reinterpret_cast<const struct sockaddr *>(&_httpAddr),(_basePath+"/"+n).c_str(),reqHeaders,obj.data(),(unsigned long)obj.length(),headers,body);
+		return (sc == 200);
+	} else {
+		const std::string path(_genPath(n,true));
+		if (!path.length())
+			return false;
+		return OSUtils::writeFile(path.c_str(),obj);
+	}
 }
 
 bool JSONDB::put(const std::string &n,const nlohmann::json &obj)
 {
-	if (!_isValidObjectName(n))
-		return false;
-
-	const std::string path(_genPath(n,true));
-	if (!path.length())
-		return false;
-
-	const std::string buf(OSUtils::jsonDump(obj));
-	if (!OSUtils::writeFile(path.c_str(),buf))
-		return false;
-
-	_E &e = _db[n];
-	e.obj = obj;
-	e.lastModifiedOnDisk = OSUtils::getLastModified(path.c_str());
-	e.lastCheck = OSUtils::now();
-
-	return true;
+	const bool r = writeRaw(n,OSUtils::jsonDump(obj));
+	_db[n].obj = obj;
+	return r;
 }
 
-const nlohmann::json &JSONDB::get(const std::string &n,unsigned long maxSinceCheck)
+const nlohmann::json &JSONDB::get(const std::string &n)
 {
+	while (!_ready) {
+		Thread::sleep(250);
+		_ready = _reload(_basePath,std::string());
+	}
+
 	if (!_isValidObjectName(n))
 		return _EMPTY_JSON;
-
-	const uint64_t now = OSUtils::now();
-	std::string buf;
 	std::map<std::string,_E>::iterator e(_db.find(n));
-
-	if (e != _db.end()) {
-		if ((now - e->second.lastCheck) <= (uint64_t)maxSinceCheck)
-			return e->second.obj;
-
-		const std::string path(_genPath(n,false));
-		if (!path.length()) // sanity check
-			return _EMPTY_JSON;
-
-		// We are somewhat tolerant to momentary disk failures here. This may
-		// occur over e.g. EC2's elastic filesystem (NFS).
-		const uint64_t lm = OSUtils::getLastModified(path.c_str());
-		if (e->second.lastModifiedOnDisk != lm) {
-			if (OSUtils::readFile(path.c_str(),buf)) {
-				try {
-					e->second.obj = OSUtils::jsonParse(buf);
-					e->second.lastModifiedOnDisk = lm; // don't update these if there is a parse error -- try again and again ASAP
-					e->second.lastCheck = now;
-				} catch ( ... ) {} // parse errors result in "holding pattern" behavior
-			}
-		}
-
+	if (e != _db.end())
 		return e->second.obj;
+
+	std::string buf;
+	if (_httpAddr) {
+		std::map<std::string,std::string> headers;
+		const unsigned int sc = Http::GET(1048576,ZT_JSONDB_HTTP_TIMEOUT,reinterpret_cast<const struct sockaddr *>(&_httpAddr),(_basePath+"/"+n).c_str(),_ZT_JSONDB_GET_HEADERS,headers,buf);
+		if (sc != 200)
+			return _EMPTY_JSON;
 	} else {
 		const std::string path(_genPath(n,false));
 		if (!path.length())
 			return _EMPTY_JSON;
-
 		if (!OSUtils::readFile(path.c_str(),buf))
 			return _EMPTY_JSON;
+	}
 
-		const uint64_t lm = OSUtils::getLastModified(path.c_str());
+	try {
 		_E &e2 = _db[n];
-		try {
-			e2.obj = OSUtils::jsonParse(buf);
-		} catch ( ... ) {
-			e2.obj = _EMPTY_JSON;
-			buf = "{}";
-		}
-		e2.lastModifiedOnDisk = lm;
-		e2.lastCheck = now;
-
+		e2.obj = OSUtils::jsonParse(buf);
 		return e2.obj;
+	} catch ( ... ) {
+		_db.erase(n);
+		return _EMPTY_JSON;
 	}
 }
 
@@ -118,23 +124,52 @@ void JSONDB::erase(const std::string &n)
 	if (!_isValidObjectName(n))
 		return;
 
-	std::string path(_genPath(n,true));
-	if (!path.length())
-		return;
+	if (_httpAddr) {
+		std::string body;
+		std::map<std::string,std::string> headers;
+		Http::DEL(1048576,ZT_JSONDB_HTTP_TIMEOUT,reinterpret_cast<const struct sockaddr *>(&_httpAddr),(_basePath+"/"+n).c_str(),_ZT_JSONDB_GET_HEADERS,headers,body);
+	} else {
+		std::string path(_genPath(n,true));
+		if (!path.length())
+			return;
+		OSUtils::rm(path.c_str());
+	}
 
-	OSUtils::rm(path.c_str());
 	_db.erase(n);
 }
 
-void JSONDB::_reload(const std::string &p,const std::string &b)
+bool JSONDB::_reload(const std::string &p,const std::string &b)
 {
-	std::vector<std::string> dl(OSUtils::listDirectory(p.c_str()));
-	for(std::vector<std::string>::const_iterator di(dl.begin());di!=dl.end();++di) {
-		if ((di->length() > 5)&&(di->substr(di->length() - 5) == ".json")) {
-			this->get(b + di->substr(0,di->length() - 5),0);
-		} else {
-			this->_reload((p + ZT_PATH_SEPARATOR + *di),(b + *di + ZT_PATH_SEPARATOR));
+	if (_httpAddr) {
+		std::string body;
+		std::map<std::string,std::string> headers;
+		const unsigned int sc = Http::GET(2147483647,ZT_JSONDB_HTTP_TIMEOUT,reinterpret_cast<const struct sockaddr *>(&_httpAddr),_basePath.c_str(),_ZT_JSONDB_GET_HEADERS,headers,body);
+		if (sc == 200) {
+			try {
+				nlohmann::json dbImg(OSUtils::jsonParse(body));
+				std::string tmp;
+				if (dbImg.is_object()) {
+					for(nlohmann::json::iterator i(dbImg.begin());i!=dbImg.end();++i) {
+						if (i.value().is_object()) {
+							tmp = i.key();
+							_db[tmp].obj = i.value();
+						}
+					}
+					return true;
+				}
+			} catch ( ... ) {} // invalid JSON, so maybe incomplete request
 		}
+		return false;
+	} else {
+		std::vector<std::string> dl(OSUtils::listDirectory(p.c_str(),true));
+		for(std::vector<std::string>::const_iterator di(dl.begin());di!=dl.end();++di) {
+			if ((di->length() > 5)&&(di->substr(di->length() - 5) == ".json")) {
+				this->get(b + di->substr(0,di->length() - 5));
+			} else {
+				this->_reload((p + ZT_PATH_SEPARATOR + *di),(b + *di + ZT_PATH_SEPARATOR));
+			}
+		}
+		return true;
 	}
 }
 
@@ -158,15 +193,23 @@ std::string JSONDB::_genPath(const std::string &n,bool create)
 	if (pt.size() == 0)
 		return std::string();
 
+	char sep;
+	if (_httpAddr) {
+		sep = '/';
+		create = false;
+	} else {
+		sep = ZT_PATH_SEPARATOR;
+	}
+
 	std::string p(_basePath);
 	if (create) OSUtils::mkdir(p.c_str());
 	for(unsigned long i=0,j=(unsigned long)(pt.size()-1);i<j;++i) {
-		p.push_back(ZT_PATH_SEPARATOR);
+		p.push_back(sep);
 		p.append(pt[i]);
 		if (create) OSUtils::mkdir(p.c_str());
 	}
 
-	p.push_back(ZT_PATH_SEPARATOR);
+	p.push_back(sep);
 	p.append(pt[pt.size()-1]);
 	p.append(".json");
 

controller/JSONDB.hpp

@@ -31,28 +31,22 @@
 
 #include "../node/Constants.hpp"
 #include "../node/Utils.hpp"
+#include "../node/InetAddress.hpp"
+#include "../node/Mutex.hpp"
 #include "../ext/json/json.hpp"
 #include "../osdep/OSUtils.hpp"
+#include "../osdep/Http.hpp"
+#include "../osdep/Thread.hpp"
 
 namespace ZeroTier {
 
 /**
- * Hierarchical JSON store that persists into the filesystem
+ * Hierarchical JSON store that persists into the filesystem or via HTTP
  */
 class JSONDB
 {
 public:
-	JSONDB(const std::string &basePath) :
-		_basePath(basePath)
-	{
-		_reload(_basePath,std::string());
-	}
-
-	inline void reload()
-	{
-		_db.clear();
-		_reload(_basePath,std::string());
-	}
+	JSONDB(const std::string &basePath);
 
 	bool writeRaw(const std::string &n,const std::string &obj);
 
@@ -63,12 +57,12 @@ public:
 	inline bool put(const std::string &n1,const std::string &n2,const std::string &n3,const std::string &n4,const nlohmann::json &obj) { return this->put((n1 + "/" + n2 + "/" + n3 + "/" + n4),obj); }
 	inline bool put(const std::string &n1,const std::string &n2,const std::string &n3,const std::string &n4,const std::string &n5,const nlohmann::json &obj) { return this->put((n1 + "/" + n2 + "/" + n3 + "/" + n4 + "/" + n5),obj); }
 
-	const nlohmann::json &get(const std::string &n,unsigned long maxSinceCheck = 0);
+	const nlohmann::json &get(const std::string &n);
 
-	inline const nlohmann::json &get(const std::string &n1,const std::string &n2,unsigned long maxSinceCheck = 0) { return this->get((n1 + "/" + n2),maxSinceCheck); }
-	inline const nlohmann::json &get(const std::string &n1,const std::string &n2,const std::string &n3,unsigned long maxSinceCheck = 0) { return this->get((n1 + "/" + n2 + "/" + n3),maxSinceCheck); }
-	inline const nlohmann::json &get(const std::string &n1,const std::string &n2,const std::string &n3,const std::string &n4,unsigned long maxSinceCheck = 0) { return this->get((n1 + "/" + n2 + "/" + n3 + "/" + n4),maxSinceCheck); }
-	inline const nlohmann::json &get(const std::string &n1,const std::string &n2,const std::string &n3,const std::string &n4,const std::string &n5,unsigned long maxSinceCheck = 0) { return this->get((n1 + "/" + n2 + "/" + n3 + "/" + n4 + "/" + n5),maxSinceCheck); }
+	inline const nlohmann::json &get(const std::string &n1,const std::string &n2) { return this->get((n1 + "/" + n2)); }
+	inline const nlohmann::json &get(const std::string &n1,const std::string &n2,const std::string &n3) { return this->get((n1 + "/" + n2 + "/" + n3)); }
+	inline const nlohmann::json &get(const std::string &n1,const std::string &n2,const std::string &n3,const std::string &n4) { return this->get((n1 + "/" + n2 + "/" + n3 + "/" + n4)); }
+	inline const nlohmann::json &get(const std::string &n1,const std::string &n2,const std::string &n3,const std::string &n4,const std::string &n5) { return this->get((n1 + "/" + n2 + "/" + n3 + "/" + n4 + "/" + n5)); }
 
 	void erase(const std::string &n);
 
@@ -78,11 +72,16 @@ public:
 	inline void erase(const std::string &n1,const std::string &n2,const std::string &n3,const std::string &n4,const std::string &n5) { this->erase(n1 + "/" + n2 + "/" + n3 + "/" + n4 + "/" + n5); }
 
 	template<typename F>
-	inline void filter(const std::string &prefix,unsigned long maxSinceCheck,F func)
+	inline void filter(const std::string &prefix,F func)
 	{
+		while (!_ready) {
+			Thread::sleep(250);
+			_ready = _reload(_basePath,std::string());
+		}
+
 		for(std::map<std::string,_E>::iterator i(_db.lower_bound(prefix));i!=_db.end();) {
 			if ((i->first.length() >= prefix.length())&&(!memcmp(i->first.data(),prefix.data(),prefix.length()))) {
-				if (!func(i->first,get(i->first,maxSinceCheck))) {
+				if (!func(i->first,get(i->first))) {
 					std::map<std::string,_E>::iterator i2(i); ++i2;
 					this->erase(i->first);
 					i = i2;
@@ -95,22 +94,21 @@ public:
 	inline bool operator!=(const JSONDB &db) const { return (!(*this == db)); }
 
 private:
-	void _reload(const std::string &p,const std::string &b);
+	bool _reload(const std::string &p,const std::string &b);
 	bool _isValidObjectName(const std::string &n);
 	std::string _genPath(const std::string &n,bool create);
 
 	struct _E
 	{
 		nlohmann::json obj;
-		uint64_t lastModifiedOnDisk;
-		uint64_t lastCheck;
-
 		inline bool operator==(const _E &e) const { return (obj == e.obj); }
 		inline bool operator!=(const _E &e) const { return (obj != e.obj); }
 	};
 
+	InetAddress _httpAddr;
 	std::string _basePath;
 	std::map<std::string,_E> _db;
+	volatile bool _ready;
 };
 
 } // namespace ZeroTier

debian/changelog

@@ -1,3 +1,9 @@
+zerotier-one (1.2.4) unstable; urgency=medium
+
+  * See https://github.com/zerotier/ZeroTierOne for release notes.
+
+ -- Adam Ierymenko <adam.ierymenko@zerotier.com>  Mon, 24 Mar 2017 01:00:00 -0700
+
 zerotier-one (1.2.2) unstable; urgency=medium
 
   * See https://github.com/zerotier/ZeroTierOne for release notes.

debian/control

@@ -10,7 +10,7 @@ Homepage: https://www.zerotier.com/
 
 Package: zerotier-one
 Architecture: any
-Depends:  ${shlibs:Depends}, ${misc:Depends}, iproute2, adduser
+Depends:  ${shlibs:Depends}, ${misc:Depends}, iproute2, adduser, libstdc++6
 Homepage: https://www.zerotier.com/
 Description: ZeroTier network virtualization service
  ZeroTier One lets you join ZeroTier virtual networks and

debian/control.wheezy

@@ -10,7 +10,7 @@ Homepage: https://www.zerotier.com/
 
 Package: zerotier-one
 Architecture: any
-Depends:  ${shlibs:Depends}, ${misc:Depends}, iproute
+Depends:  ${shlibs:Depends}, ${misc:Depends}, iproute, libstdc++6
 Homepage: https://www.zerotier.com/
 Description: ZeroTier network virtualization service
  ZeroTier One lets you join ZeroTier virtual networks and

ext/arm32-neon-salsa2012-asm/README.md

@@ -0,0 +1,6 @@
+ARM NEON (32-bit) ASM implementation of Salsa20/12
+======
+
+This is from [supercop](http://bench.cr.yp.to/supercop.html) and was originally written by Daniel J. Bernstein. Code is in the public domain like the rest of Salsa20. It's much faster than the naive implementation.
+
+It's included automatically in 32-bit Linux ARM builds. It likely will not work on 64-bit ARM, so it'll need to be ported at least. That will unfortunately keep it out of mobile versions for now since those are all going 64-bit.

ext/arm32-neon-salsa2012-asm/salsa2012.h

@@ -0,0 +1,23 @@
+#ifndef ZT_SALSA2012_ARM32NEON_ASM
+#define ZT_SALSA2012_ARM32NEON_ASM
+
+#if defined(__linux__) || defined(linux) || defined(__LINUX__) || defined(__linux)
+#include <sys/auxv.h>
+#include <asm/hwcap.h>
+#define zt_arm_has_neon() ((getauxval(AT_HWCAP) & HWCAP_NEON) != 0)
+#else
+#define zt_arm_has_neon() (true)
+#endif
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+// ciphertext buffer, message/NULL, length, nonce (8 bytes), key (32 bytes)
+extern int zt_salsa2012_armneon3_xor(unsigned char *c,const unsigned char *m,unsigned long long len,const unsigned char *n,const unsigned char *k);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif

ext/installfiles/mac/ZeroTier One.pkgproj

@@ -646,7 +646,7 @@
 			<key>OVERWRITE_PERMISSIONS</key>
 			<false/>
 			<key>VERSION</key>
-			<string>1.2.2</string>
+			<string>1.2.4</string>
 		</dict>
 		<key>PROJECT_COMMENTS</key>
 		<dict>

ext/installfiles/mac/uninstall.sh

@@ -28,13 +28,11 @@ echo "Removing ZeroTier One files..."
 
 rm -rf '/Applications/ZeroTier One.app'
 rm -f '/usr/bin/zerotier-one' '/usr/bin/zerotier-idtool' '/usr/bin/zerotier-cli' '/Library/LaunchDaemons/com.zerotier.one.plist'
-mkdir -p /tmp/ZeroTierOne_uninstall_tmp
-cp "/Library/Application Support/ZeroTier/One/*.secret" /tmp/ZeroTierOne_uninstall_tmp
-rm -rf '/Library/Application Support/ZeroTier/One'
-mkdir -p '/Library/Application Support/ZeroTier/One'
-cp "/tmp/ZeroTierOne_uninstall_tmp/*.secret" '/Library/Application Support/ZeroTier/One'
-chmod 0600 "/Library/Application Support/ZeroTier/One/*.secret"
-rm -rf /tmp/ZeroTierOne_uninstall_tmp
+
+cd '/Library/Application Support/ZeroTier/One'
+if [ "`pwd`" = '/Library/Application Support/ZeroTier/One' ]; then
+	rm -rf *.d *.sh *.log *.old *.kext *.conf *.pkg *.dmg *.pid *.port *.save *.bin planet zerotier-* devicemap
+fi
 
 echo 'Uninstall complete.'
 echo

ext/installfiles/windows/ZeroTier One.aip

@@ -27,10 +27,10 @@
     <ROW Property="CTRLS" Value="2"/>
     <ROW Property="MSIFASTINSTALL" MultiBuildValue="DefaultBuild:2"/>
     <ROW Property="Manufacturer" Value="ZeroTier, Inc."/>
-    <ROW Property="ProductCode" Value="1033:{C2C8F21B-95CA-4E32-90E1-154D180212FE} " Type="16"/>
+    <ROW Property="ProductCode" Value="1033:{0AF2DC73-6629-41A1-B0C6-B110582FEE29} " Type="16"/>
     <ROW Property="ProductLanguage" Value="1033"/>
     <ROW Property="ProductName" Value="ZeroTier One"/>
-    <ROW Property="ProductVersion" Value="1.2.2" Type="32"/>
+    <ROW Property="ProductVersion" Value="1.2.4" Type="32"/>
     <ROW Property="REBOOT" MultiBuildValue="DefaultBuild:ReallySuppress"/>
     <ROW Property="RUNAPPLICATION" Value="1" Type="4"/>
     <ROW Property="SecureCustomProperties" Value="OLDPRODUCTS;AI_NEWERPRODUCTFOUND;AI_SETUPEXEPATH;SETUPEXEDIR"/>
@@ -60,7 +60,7 @@
     <ROW Directory="x86_Dir" Directory_Parent="tapwindows_Dir" DefaultDir="x86"/>
   </COMPONENT>
   <COMPONENT cid="caphyon.advinst.msicomp.MsiCompsComponent">
-    <ROW Component="AI_CustomARPName" ComponentId="{B16DE57C-0F91-4A6D-B059-E2467CACEC1C}" Directory_="APPDIR" Attributes="4" KeyPath="DisplayName" Options="1"/>
+    <ROW Component="AI_CustomARPName" ComponentId="{BEF49A0A-3D61-4162-8934-88473D8A6F3D}" Directory_="APPDIR" Attributes="4" KeyPath="DisplayName" Options="1"/>
     <ROW Component="AI_DisableModify" ComponentId="{020DCABD-5D56-49B9-AF48-F07F0B55E590}" Directory_="APPDIR" Attributes="4" KeyPath="NoModify" Options="1"/>
     <ROW Component="AI_ExePath" ComponentId="{8E02B36C-7A19-429B-A93E-77A9261AC918}" Directory_="APPDIR" Attributes="4" KeyPath="AI_ExePath"/>
     <ROW Component="Hardcodet.Wpf.TaskbarNotification.dll" ComponentId="{BEA825AF-2555-44AF-BE40-47FFC16DCBA6}" Directory_="APPDIR" Attributes="0" KeyPath="Hardcodet.Wpf.TaskbarNotification.dll"/>
@@ -432,7 +432,7 @@
     <ROW XmlAttribute="xsischemaLocation" XmlElement="swidsoftware_identification_tag" Name="xsi:schemaLocation" Flags="14" Order="3" Value="http://standards.iso.org/iso/19770/-2/2008/schema.xsd software_identification_tag.xsd"/>
   </COMPONENT>
   <COMPONENT cid="caphyon.advinst.msicomp.XmlElementComponent">
-    <ROW XmlElement="swidbuild" ParentElement="swidnumeric" Name="swid:build" Condition="1" Order="2" Flags="14" Text="2"/>
+    <ROW XmlElement="swidbuild" ParentElement="swidnumeric" Name="swid:build" Condition="1" Order="2" Flags="14" Text="4"/>
     <ROW XmlElement="swidentitlement_required_indicator" ParentElement="swidsoftware_identification_tag" Name="swid:entitlement_required_indicator" Condition="1" Order="0" Flags="14" Text="false"/>
     <ROW XmlElement="swidmajor" ParentElement="swidnumeric" Name="swid:major" Condition="1" Order="0" Flags="14" Text="1"/>
     <ROW XmlElement="swidminor" ParentElement="swidnumeric" Name="swid:minor" Condition="1" Order="1" Flags="14" Text="2"/>

ext/installfiles/windows/chocolatey/zerotier-one/zerotier-one.nuspec

@@ -26,7 +26,7 @@ This is a nuspec. It mostly adheres to https://docs.nuget.org/create/Nuspec-Refe
     <!-- version should MATCH as closely as possible with the underlying software -->
     <!-- Is the version a prerelease of a version? https://docs.nuget.org/create/versioning#creating-prerelease-packages -->
     <!-- Note that unstable versions like 0.0.1 can be considered a released version, but it's possible that one can release a 0.0.1-beta before you release a 0.0.1 version. If the version number is final, that is considered a released version and not a prerelease. -->
-    <version>1.2.2</version>
+    <version>1.2.4</version>
     <!-- <packageSourceUrl>Where is this Chocolatey package located (think GitHub)? packageSourceUrl is highly recommended for the community feed</packageSourceUrl>-->
     <!-- owners is a poor name for maintainers of the package. It sticks around by this name for compatibility reasons. It basically means you. -->
     <!--<owners>ZeroTier, Inc.</owners>-->

ext/x64-salsa2012-asm/README.md

@@ -0,0 +1,6 @@
+Blazingly fast X64 ASM implementation of Salsa20/12
+======
+
+This is ripped from the [cnacl](https://github.com/cjdelisle/cnacl) source. The actual code is by Danial J. Bernstein and is in the public domain.
+
+This is included on Linux and Mac 64-bit builds and is significantly faster than the SSE intrinsics or C versions. It's used for packet encode/decode only since its use differs a bit from the regular Salsa20 C++ class. Specifically it lacks the ability to be called on multiple blocks, preferring instead to take a key and a single stream to encrypt and that's it.

ext/x64-salsa2012-asm/salsa2012.h

@@ -0,0 +1,16 @@
+#ifndef ZT_X64_SALSA2012_ASM
+#define ZT_X64_SALSA2012_ASM
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+// Generates Salsa20/12 key stream
+// output, outlen, nonce, key (256-bit / 32-byte)
+extern int zt_salsa2012_amd64_xmm6(unsigned char *, unsigned long long, const unsigned char *, const unsigned char *);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif