0.3.0: BREAKS PROTOCOL BACKWARD COMPATIBILITY

This version is not compatible with versions prior to 0.3.0, so
'git pull' and restart if you are following the alpha.

Changes from 0.2.5:

 - All multicast frames are now signed by the original sender. This
   will permit very efficient and fault tolerant rate limitation
   across networks, and imposes a kind of "hash cash" cost on those
   who wish to flood the network by forcing them to keep regenerating
   new identities.

 - Simplified peer last unicast / last multicast accounting.

 - Improvements to multicast propagation algorithm to achieve better
   coverage with less redundant messages.

 - The bloated Switch class went on a diet, having packet decoding
   broken out into PacketDecoder and multicast propagation algorithm
   broken out into Multicaster.

 - Multicaster is implemented as a template mockable class to permit
   future simulations of huge scale multicast using the actual code
   instead of mockups in another language.

 - Introduced a faster non-cryptographic random source for things
   like multicast propagation and address choosing.

 - Some code cleanup, removal of outdated comments, etc.

README.md

@@ -50,9 +50,6 @@ Check out the [blog](http://blog.zerotier.com/) for announcements, in-depth arti
 **Q:** What about privacy? Does this hide my location on the network?  
 **A:** ZeroTier is not a connection anonymizer. Other than encryption, it doesn't do anything special to hide your identity or network location. If you want strong privacy protection there are already very advanced tools like [Tor](https://www.torproject.org) for that, and this isn't trying to duplicate their functionality. At the same time, ZT does not do anything special to harm your privacy either. It's not spyware or snoop-ware.
 
-**Q:** You say "almost unlimited size." Isn't multicast and broadcast traffic eventually going to be too much? What happens then?  
-**A:** ZeroTier One uses an algorithm called *implicit social switching*. The overall maximum number of recipients for a multicast is limited, so if there are too many listeners to a given multicast address then obviously not everyone will receive every message. So who does? Social switching causes multicasts to propagate in the direction of peers to whom you have recently communicated. As a result, multicasts tend to propagate along lines of association. The people most likely to get your service announcements are those with whom you frequently connect.
-
 **Q:** Is this designed to replace IP, BGP, IPv6, routers, etc.?  
 **A:** No. Its purpose is to act as a collaboration tool, a VPN alternative, a network mobility tool, a testbed for the development of software that takes advantage of fully open networking, a virtual LAN party for gamers, and so on, but it's not intended (or able) to replace the core of the Internet.
 
@@ -92,6 +89,7 @@ Check out the [blog](http://blog.zerotier.com/) for announcements, in-depth arti
 
 ----
 
-<a href="http://flattr.com/thing/1611614/ZeroTier-Networks" target="_blank"><img src="http://api.flattr.com/button/flattr-badge-large.png" alt="Flattr this" title="Flattr this" border="0" /></a>
+<a href="http://flattr.com/thing/1611614/ZeroTier-Networks" target="_blank"><img src="http://api.flattr.com/button/flattr-badge-large.png" alt="Flattr this" title="Flattr this" border="0" /></a>  
+[![githalytics.com alpha](https://cruel-carlota.pagodabox.com/59b2cbb9c154bf84bddb4b714402e548 "githalytics.com")](http://githalytics.com/zerotier/ZeroTierOne)
 
 (c)2012-2013 [ZeroTier Networks LLC](https://www.zerotier.com/)

node/BloomFilter.hpp

@@ -0,0 +1,133 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_BLOOMFILTER_HPP
+#define _ZT_BLOOMFILTER_HPP
+
+#include <string.h>
+#include "Utils.hpp"
+
+namespace ZeroTier {
+
+/**
+ * A simple bit field bloom filter
+ *
+ * This actually isn't a total filter, in that it does not contain a hashing
+ * algorithm. It's up to the caller to hash/sum the items being remembered
+ * so that the distribution of 'n' is random.
+ *
+ * @tparam B Size in BITS, must be a multiple of 8
+ */
+template<unsigned int B>
+class BloomFilter
+{
+public:
+	/**
+	 * Construct an empty filter
+	 */
+	BloomFilter()
+		throw()
+	{
+		memset(_field,0,sizeof(_field));
+	}
+
+	/**
+	 * Construct from a raw filter
+	 *
+	 * @param b Raw filter bits, must be exactly bytes() in length, or NULL to construct empty
+	 */
+	BloomFilter(const void *b)
+		throw()
+	{
+		if (b)
+			memcpy(_field,b,sizeof(_field));
+		else memset(_field,0,sizeof(_field));
+	}
+
+	/**
+	 * @return Size of filter in bits
+	 */
+	static inline unsigned int bits() throw() { return B; }
+
+	/**
+	 * @return Size of filter in bytes
+	 */
+	static inline unsigned int bytes() throw() { return (B / 8); }
+
+	/**
+	 * @return Pointer to portable array of bytes of bytes() length representing filter
+	 */
+	inline const unsigned char *data() const throw() { return _field; }
+
+	/**
+	 * Clear all bits in filter
+	 */
+	inline void clear()
+		throw()
+	{
+		memset(_field,0,sizeof(_field));
+	}
+
+	/**
+	 * @param n Value to set
+	 */
+	inline void set(unsigned int n)
+		throw()
+	{
+		n %= B;
+		_field[n / 8] |= (1 << (n % 8));
+	}
+
+	/**
+	 * @param n Value to test
+	 * @return True if value is present is set
+	 */
+	inline bool contains(unsigned int n)
+		throw()
+	{
+		n %= B;
+		return (_field[n / 8] & (1 << (n % 8)));
+	}
+
+	/**
+	 * Clear a random bit in this bloom filter
+	 *
+	 * @param rn Random number
+	 */
+	inline void decay(unsigned int rn)
+		throw()
+	{
+		_field[(rn >> 3) % (B / 8)] &= ~((unsigned char)(1 << (rn & 7)));
+	}
+
+private:
+	unsigned char _field[B / 8];
+};
+
+} // namespace ZeroTier
+
+#endif

node/Buffer.hpp

@@ -240,6 +240,22 @@ public:
 		_l += sizeof(T);
 	}
 
+	/**
+	 * Append a run of bytes
+	 *
+	 * @param c Character value to append
+	 * @param n Number of times to append
+	 * @throws std::out_of_range Attempt to append beyond capacity
+	 */
+	inline void append(unsigned char c,unsigned int n)
+		throw(std::out_of_range)
+	{
+		if ((_l + n) > C)
+			throw std::out_of_range("Buffer: append beyond capacity");
+		for(unsigned int i=0;i<n;++i)
+			_b[_l++] = (char)c;
+	}
+
 	/**
 	 * Append a C-array of bytes
 	 * 

node/CMWC4096.hpp

@@ -0,0 +1,91 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_CMWC4096_HPP
+#define _ZT_CMWC4096_HPP
+
+#include <stdint.h>
+#include "Utils.hpp"
+
+namespace ZeroTier {
+
+/** 
+ * Complement Multiply With Carry random number generator
+ *
+ * Based on original code posted to Usenet in the public domain by
+ * George Marsaglia. Period is approximately 2^131086.
+ *
+ * This is not used for cryptographic purposes but for a very fast
+ * and high-quality PRNG elsewhere in the code.
+ */
+class CMWC4096
+{
+public:
+	/**
+	 * Construct and initialize from secure random source
+	 */
+	CMWC4096()
+		throw()
+	{
+		Utils::getSecureRandom(Q,sizeof(Q));
+		Utils::getSecureRandom(&c,sizeof(c));
+		c %= 809430660;
+		i = 4095;
+	}
+
+	inline uint32_t next32()
+		throw()
+	{
+		uint32_t __i = ++i & 4095;
+		const uint64_t t = (18782ULL * (uint64_t)Q[__i]) + (uint64_t)c;
+		c = (uint32_t)(t >> 32);
+		uint32_t x = c + (uint32_t)t;
+		const uint32_t p = (uint32_t)(x < c); x += p; c += p;
+		return (Q[__i] = 0xfffffffe - x);
+	}
+
+	inline uint64_t next64()
+		throw()
+	{
+		return ((((uint64_t)next32()) << 32) ^ (uint64_t)next32());
+	}
+
+	inline double nextDouble()
+		throw()
+	{
+		return ((double)(next32()) / 4294967296.0);
+	}
+
+private:
+	uint32_t Q[4096];
+	uint32_t c;
+	uint32_t i;
+};
+
+} // namespace ZeroTier
+
+#endif

node/Constants.hpp

@@ -235,12 +235,17 @@ error_no_ZT_ARCH_defined;
 /**
  * Length of circular ring buffer history of multicast packets
  */
-#define ZT_MULTICAST_DEDUP_HISTORY_LENGTH 4096
+#define ZT_MULTICAST_DEDUP_HISTORY_LENGTH 1024
 
 /**
  * Expiration time in ms for multicast history items
  */
-#define ZT_MULTICAST_DEDUP_HISTORY_EXPIRE 8000
+#define ZT_MULTICAST_DEDUP_HISTORY_EXPIRE 4000
+
+/**
+ * Number of bits to randomly "decay" in bloom filter per hop
+ */
+#define ZT_MULTICAST_BLOOM_FILTER_DECAY_RATE 2
 
 /**
  * Period between announcements of all multicast 'likes' in ms

node/Demarc.cpp

@@ -143,7 +143,7 @@ Demarc::Port Demarc::pick(const InetAddress &to) const
 			}
 		}
 		if (possibilities.size())
-			return possibilities[Utils::randomInt<unsigned int>() % possibilities.size()]->first;
+			return possibilities[_r->prng->next32() % possibilities.size()]->first;
 		else return NULL_PORT;
 	} catch ( ... ) {
 		return NULL_PORT;
@@ -174,7 +174,7 @@ Demarc::Port Demarc::send(Demarc::Port fromPort,const InetAddress &to,const void
 				}
 			}
 			if (possibilities.size())
-				pe = possibilities[Utils::randomInt<unsigned int>() % possibilities.size()];
+				pe = possibilities[_r->prng->next32() % possibilities.size()];
 			else {
 				_ports_m.unlock();
 				return NULL_PORT;

node/Http.cpp

@@ -173,7 +173,7 @@ void Http::Request::main()
 				addrList->sort();
 				addrList->unique();
 				unsigned int i = 0,k = 0;
-				k = Utils::randomInt<unsigned int>() % addrList->size();
+				k = rand() % addrList->size();
 				std::list<InetAddress>::iterator a(addrList->begin());
 				while (i++ != k) ++a;
 				addr = &(*a);

node/Multicaster.hpp

@@ -0,0 +1,325 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_MULTICASTER_HPP
+#define _ZT_MULTICASTER_HPP
+
+#include <stdint.h>
+#include <string.h>
+
+#include <openssl/sha.h>
+
+#include <utility>
+#include <algorithm>
+#include <map>
+#include <set>
+#include <vector>
+#include <string>
+
+#include "Constants.hpp"
+#include "Buffer.hpp"
+#include "Packet.hpp"
+#include "MulticastGroup.hpp"
+#include "Utils.hpp"
+#include "MAC.hpp"
+#include "Address.hpp"
+#include "SharedPtr.hpp"
+#include "BloomFilter.hpp"
+#include "Identity.hpp"
+#include "CMWC4096.hpp"
+
+// Maximum sample size to pick during choice of multicast propagation peers
+#define ZT_MULTICAST_PICK_MAX_SAMPLE_SIZE (ZT_MULTICAST_PROPAGATION_BREADTH * 8)
+
+namespace ZeroTier {
+
+/**
+ * Multicast propagation engine
+ *
+ * This is written as a generic class so that it can be mocked and tested
+ * in simulation. It also always takes 'now' as an argument, permitting
+ * running in simulated time.
+ */
+class Multicaster
+{
+public:
+	/**
+	 * 256-bit simple bloom filter included with multicast frame packets
+	 */
+	typedef BloomFilter<ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE_BITS> MulticastBloomFilter;
+
+	Multicaster()
+		throw()
+	{
+		memset(_multicastHistory,0,sizeof(_multicastHistory));
+		_multicastHistoryPtr = 0;
+	}
+
+	/**
+	 * Generate a signature of a multicast packet using an identity
+	 *
+	 * @param id Identity to sign with (must have secret key portion)
+	 * @param nwid Network ID
+	 * @param from MAC address of sender
+	 * @param to Multicast group
+	 * @param etherType 16-bit ethernet type
+	 * @param data Ethernet frame data
+	 * @param len Length of frame
+	 * @return ECDSA signature
+	 */
+	static inline std::string signMulticastPacket(const Identity &id,uint64_t nwid,const MAC &from,const MulticastGroup &to,unsigned int etherType,const void *data,unsigned int len)
+	{
+		unsigned char digest[32];
+		_hashMulticastPacketForSig(nwid,from,to,etherType,data,len,digest);
+		return id.sign(digest);
+	}
+
+	/**
+	 * Verify a signature from a multicast packet
+	 *
+	 * @param id Identity of original signer
+	 * @param nwid Network ID
+	 * @param from MAC address of sender
+	 * @param to Multicast group
+	 * @param etherType 16-bit ethernet type
+	 * @param data Ethernet frame data
+	 * @param len Length of frame
+	 * @param signature ECDSA signature
+	 * @param siglen Length of signature in bytes
+	 * @return ECDSA signature
+	 */
+	static bool verifyMulticastPacket(const Identity &id,uint64_t nwid,const MAC &from,const MulticastGroup &to,unsigned int etherType,const void *data,unsigned int len,const void *signature,unsigned int siglen)
+	{
+		unsigned char digest[32];
+		_hashMulticastPacketForSig(nwid,from,to,etherType,data,len,digest);
+		return id.verifySignature(digest,signature,siglen);
+	}
+
+	/**
+	 * Compute the CRC64 code for multicast deduplication
+	 *
+	 * @param nwid Network ID
+	 * @param from Sender MAC
+	 * @param to Destination multicast group
+	 * @param etherType Ethernet frame type
+	 * @param payload Multicast frame data
+	 * @param len Length of frame
+	 */
+	static inline uint64_t computeMulticastDedupCrc(
+		uint64_t nwid,
+		const MAC &from,
+		const MulticastGroup &to,
+		unsigned int etherType,
+		const void *payload,
+		unsigned int len)
+		throw()
+	{
+		// This CRC is only used locally, so byte order issues and
+		// such don't matter. It can also be changed without protocol
+		// impact.
+		uint64_t crc = Utils::crc64(0,from.data,6);
+		crc = Utils::crc64(crc,to.mac().data,6);
+		crc ^= (uint64_t)to.adi();
+		crc ^= (uint64_t)etherType;
+		crc = Utils::crc64(crc,payload,len);
+		crc ^= nwid; // also include network ID in CRC
+		return crc;
+	}
+
+	/**
+	 * Check multicast history to see if this is a duplicate
+	 *
+	 * @param crc Multicast CRC
+	 * @param now Current time
+	 * @return True if this appears to be a duplicate to within history expiration time
+	 */
+	inline bool checkDuplicate(uint64_t crc,uint64_t now) const
+		throw()
+	{
+		for(unsigned int i=0;i<ZT_MULTICAST_DEDUP_HISTORY_LENGTH;++i) {
+			if ((_multicastHistory[i][0] == crc)&&((now - _multicastHistory[i][1]) < ZT_MULTICAST_DEDUP_HISTORY_EXPIRE))
+				return true;
+		}
+		return false;
+	}
+
+	/**
+	 * Add a multicast CRC to the multicast deduplication history
+	 *
+	 * @param crc Multicast CRC
+	 * @param now Current time
+	 */
+	inline void addToDedupHistory(uint64_t crc,uint64_t now)
+		throw()
+	{
+		unsigned int mhi = ++_multicastHistoryPtr % ZT_MULTICAST_DEDUP_HISTORY_LENGTH;
+		_multicastHistory[mhi][0] = crc;
+		_multicastHistory[mhi][1] = now;
+	}
+
+	/**
+	 * Update the most recent LIKE time for an address in a given multicast group on a given network
+	 *
+	 * @param nwid Network ID
+	 * @param mg Multicast group
+	 * @param addr Address that likes group on given network
+	 * @param now Current timestamp
+	 */
+	inline void likesMulticastGroup(const uint64_t nwid,const MulticastGroup &mg,const Address &addr,const uint64_t now)
+	{
+		Mutex::Lock _l(_multicastMemberships_m);
+		std::vector<MulticastMembership> &memberships = _multicastMemberships[MulticastChannel(nwid,mg)];
+		for(std::vector<MulticastMembership>::iterator mm(memberships.begin());mm!=memberships.end();++mm) {
+			if (mm->first == addr) {
+				mm->second = now;
+				return;
+			}
+		}
+		memberships.push_back(MulticastMembership(addr,now));
+	}
+
+	/**
+	 * Choose peers to send a propagating multicast to
+	 *
+	 * @param topology Topology object or mock thereof
+	 * @param nwid Network ID
+	 * @param mg Multicast group
+	 * @param originalSubmitter Original submitter of multicast message to network
+	 * @param upstream Address from which message originated, or null (0) address if none
+	 * @param bf Bloom filter, updated in place with sums of addresses in chosen peers and/or decay
+	 * @param max Maximum number of peers to pick
+	 * @param peers Array of objects of type P to fill with up to [max] peers
+	 * @param now Current timestamp
+	 * @return Number of peers actually stored in peers array
+	 * @tparam T Type of topology, which is Topology in running code or a mock in simulation
+	 * @tparam P Type of peers, which is SharedPtr<Peer> in running code or a mock in simulation (mock must behave like a pointer type)
+	 */
+	template<typename T,typename P>
+	inline unsigned int pickNextPropagationPeers(
+		CMWC4096 &prng,
+		T &topology,
+		uint64_t nwid,
+		const MulticastGroup &mg,
+		const Address &originalSubmitter,
+		const Address &upstream,
+		MulticastBloomFilter &bf,
+		unsigned int max,
+		P *peers,
+		uint64_t now)
+	{
+		typename std::set< P,_PeerPropagationPrioritySortOrder<P> > toConsider;
+
+		// Pick up to ZT_MULTICAST_PICK_MAX_SAMPLE_SIZE peers that have
+		// subscribed to this channel and that are not in bloom filter.
+		// Pick randomly from subscribers, but place into a set that is
+		// sorted in descending order of time of most recent unicast
+		// frame transfer. (Implicit social ordering.) Also ignore original
+		// submitter and upstream, since we know these have seen this
+		// message.
+		{
+			Mutex::Lock _l(_multicastMemberships_m);
+			std::map< MulticastChannel,std::vector<MulticastMembership> >::iterator mm(_multicastMemberships.find(MulticastChannel(nwid,mg)));
+			if ((mm != _multicastMemberships.end())&&(!mm->second.empty())) {
+				for(unsigned int stries=0;stries<ZT_MULTICAST_PICK_MAX_SAMPLE_SIZE;++stries) {
+					MulticastMembership &m = mm->second[prng.next32() % mm->second.size()];
+					if (((now - m.second) < ZT_MULTICAST_LIKE_EXPIRE)&&(!bf.contains(m.first.sum()))&&(m.first != originalSubmitter)&&(m.first != upstream)) {
+						P peer(topology.getPeer(m.first));
+						if (peer)
+							toConsider.insert(peer);
+					}
+				}
+			}
+		}
+
+		// The first peers in toConsider will be the 'best'
+		unsigned int chosen = 0;
+		for(typename std::set< P,_PeerPropagationPrioritySortOrder<P> >::iterator i(toConsider.begin());((i!=toConsider.end())&&(chosen < max));++i)
+			bf.set((peers[chosen++] = *i)->address().sum());
+
+		// Add a supernode if there are fewer than the desired
+		// number of recipients.
+		if (chosen < max) {
+			P peer = topology.getBestSupernode(&originalSubmitter,1,true);
+			if (peer)
+				peers[chosen++] = peer;
+		}
+
+		return chosen;
+	}
+
+private:
+	// Sort order for chosen propagation peers
+	template<typename P>
+	struct _PeerPropagationPrioritySortOrder
+	{
+		inline bool operator()(const P &p1,const P &p2) const
+		{
+			return (p1->lastUnicastFrame() > p2->lastUnicastFrame());
+		}
+	};
+
+	static inline void _hashMulticastPacketForSig(uint64_t nwid,const MAC &from,const MulticastGroup &to,unsigned int etherType,const void *data,unsigned int len,unsigned char *digest)
+		throw()
+	{
+		unsigned char zero = 0;
+		SHA256_CTX sha;
+		SHA256_Init(&sha);
+		uint64_t _nwid = Utils::hton(nwid);
+		SHA256_Update(&sha,(unsigned char *)&_nwid,sizeof(_nwid));
+		SHA256_Update(&sha,&zero,1);
+		SHA256_Update(&sha,(unsigned char *)from.data,6);
+		SHA256_Update(&sha,&zero,1);
+		SHA256_Update(&sha,(unsigned char *)to.mac().data,6);
+		SHA256_Update(&sha,&zero,1);
+		uint32_t _adi = Utils::hton(to.adi());
+		SHA256_Update(&sha,(unsigned char *)&_adi,sizeof(_adi));
+		SHA256_Update(&sha,&zero,1);
+		uint16_t _etype = Utils::hton((uint16_t)etherType);
+		SHA256_Update(&sha,(unsigned char *)&_etype,sizeof(_etype));
+		SHA256_Update(&sha,&zero,1);
+		SHA256_Update(&sha,(unsigned char *)data,len);
+		SHA256_Final(digest,&sha);
+	}
+
+	// ring buffer: [0] - CRC, [1] - timestamp
+	uint64_t _multicastHistory[ZT_MULTICAST_DEDUP_HISTORY_LENGTH][2];
+	volatile unsigned int _multicastHistoryPtr;
+
+	// A multicast channel, essentially a pub/sub channel. It consists of a
+	// network ID and a multicast group within that network.
+	typedef std::pair<uint64_t,MulticastGroup> MulticastChannel;
+
+	// Address and time of last LIKE
+	typedef std::pair<Address,uint64_t> MulticastMembership;
+
+	std::map< MulticastChannel,std::vector<MulticastMembership> > _multicastMemberships;
+	Mutex _multicastMemberships_m;
+};
+
+} // namespace ZeroTier
+
+#endif

node/Network.hpp

@@ -42,6 +42,7 @@
 #include "RuntimeEnvironment.hpp"
 #include "MulticastGroup.hpp"
 #include "NonCopyable.hpp"
+#include "MAC.hpp"
 
 namespace ZeroTier {
 

node/Node.cpp

@@ -64,6 +64,8 @@
 #include "Network.hpp"
 #include "MulticastGroup.hpp"
 #include "Mutex.hpp"
+#include "Multicaster.hpp"
+#include "CMWC4096.hpp"
 
 #include "../version.h"
 
@@ -116,8 +118,10 @@ Node::~Node()
 	delete impl->renv.sysEnv;
 	delete impl->renv.topology;
 	delete impl->renv.sw;
+	delete impl->renv.multicaster;
 	delete impl->renv.demarc;
 	delete impl->renv.nc;
+	delete impl->renv.prng;
 	delete impl->renv.log;
 
 	delete impl;
@@ -151,6 +155,8 @@ Node::ReasonForTermination Node::run()
 
 		TRACE("initializing...");
 
+		_r->prng = new CMWC4096();
+
 		if (!_r->configAuthority.fromString(_r->configAuthorityIdentityStr))
 			return impl->terminateBecause(Node::NODE_UNRECOVERABLE_ERROR,"configuration authority identity is not valid");
 
@@ -191,8 +197,11 @@ Node::ReasonForTermination Node::run()
 		std::string ovsPath(_r->homePath + ZT_PATH_SEPARATOR_S + "thisdeviceismine");
 		if (((Utils::now() - Utils::getLastModified(ovsPath.c_str())) >= ZT_OVS_GENERATE_NEW_IF_OLDER_THAN)||(!Utils::readFile(ovsPath.c_str(),_r->ownershipVerificationSecret))) {
 			_r->ownershipVerificationSecret = "";
-			for(unsigned int i=0;i<24;++i)
-				_r->ownershipVerificationSecret.push_back("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"[Utils::randomInt<unsigned int>() % 62]);
+			unsigned int securern = 0;
+			for(unsigned int i=0;i<24;++i) {
+				Utils::getSecureRandom(&securern,sizeof(securern));
+				_r->ownershipVerificationSecret.push_back("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"[securern % 62]);
+			}
 			_r->ownershipVerificationSecret.append(ZT_EOL_S);
 			if (!Utils::writeFile(ovsPath.c_str(),_r->ownershipVerificationSecret))
 				return impl->terminateBecause(Node::NODE_UNRECOVERABLE_ERROR,"could not write 'thisdeviceismine' (home path not writable?)");
@@ -211,6 +220,7 @@ Node::ReasonForTermination Node::run()
 		// watcher.
 		_r->nc = new NodeConfig(_r,_r->autoconfUrlPrefix + _r->identity.address().toString());
 		_r->demarc = new Demarc(_r);
+		_r->multicaster = new Multicaster();
 		_r->sw = new Switch(_r);
 		_r->topology = new Topology(_r,(_r->homePath + ZT_PATH_SEPARATOR_S + "peer.db").c_str());
 		_r->sysEnv = new SysEnv(_r);
@@ -396,7 +406,7 @@ Node::ReasonForTermination Node::run()
 			try {
 				unsigned long delay = std::min((unsigned long)ZT_MIN_SERVICE_LOOP_INTERVAL,_r->sw->doTimerTasks());
 				uint64_t start = Utils::now();
-				Thread::sleep(delay);
+				_r->mainLoopWaitCondition.wait(delay);
 				lastDelayDelta = (long)(Utils::now() - start) - (long)delay;
 			} catch (std::exception &exc) {
 				LOG("unexpected exception running Switch doTimerTasks: %s",exc.what());
@@ -423,12 +433,14 @@ void Node::terminate()
 	throw()
 {
 	((_NodeImpl *)_impl)->terminateNow = true;
+	((_NodeImpl *)_impl)->renv.mainLoopWaitCondition.signal();
 }
 
 void Node::updateStatusNow()
 	throw()
 {
 	((_NodeImpl *)_impl)->updateStatusNow = true;
+	((_NodeImpl *)_impl)->renv.mainLoopWaitCondition.signal();
 }
 
 class _VersionStringMaker

node/NodeConfig.cpp

@@ -152,7 +152,7 @@ void NodeConfig::__CBautoconfHandler(const std::string &lastModified,const std::
 									if (rawAddr.length() == ZT_ADDRESS_LENGTH) {
 										Address addr(rawAddr.data());
 										if ((addr)&&(!addr.isReserved())) {
-											TRACE("network %llu member: %s",nwid,addr.toString().c_str());
+											//TRACE("network %llu member: %s",nwid,addr.toString().c_str());
 											nw->_members.insert(addr);
 										}
 									}

node/Packet.hpp

@@ -45,8 +45,13 @@
 
 /**
  * Protocol version
+ *
+ * 1 - 0.2.0 ... 0.2.5
+ * 2 - 0.3.0 ...
+ *   * Added signature and originating peer to multicast frame
+ *   * Double size of multicast frame bloom filter
  */
-#define ZT_PROTO_VERSION 1
+#define ZT_PROTO_VERSION 2
 
 /**
  * Maximum hop count allowed by packet structure (3 bits, 0-7)
@@ -122,7 +127,9 @@
  */
 #define ZT_PROTO_MIN_FRAGMENT_LENGTH ZT_PACKET_FRAGMENT_IDX_PAYLOAD
 
-#define ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE 32
+// Size of bloom filter used in multicast propagation
+#define ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE_BITS 512
+#define ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE_BYTES 64
 
 // Field incides for parsing verbs
 #define ZT_PROTO_VERB_HELLO_IDX_PROTOCOL_VERSION (ZT_PACKET_IDX_PAYLOAD)
@@ -146,15 +153,18 @@
 #define ZT_PROTO_VERB_FRAME_IDX_NETWORK_ID (ZT_PACKET_IDX_PAYLOAD)
 #define ZT_PROTO_VERB_FRAME_IDX_ETHERTYPE (ZT_PROTO_VERB_FRAME_IDX_NETWORK_ID + 8)
 #define ZT_PROTO_VERB_FRAME_IDX_PAYLOAD (ZT_PROTO_VERB_FRAME_IDX_ETHERTYPE + 2)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_NETWORK_ID (ZT_PACKET_IDX_PAYLOAD)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_MULTICAST_MAC (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_NETWORK_ID + 8)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ADI (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_MULTICAST_MAC + 6)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_BLOOM (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ADI + 4)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_HOPS (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_BLOOM + ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_LOAD_FACTOR (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_HOPS + 1)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_FROM_MAC (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_LOAD_FACTOR + 2)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ETHERTYPE (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_FROM_MAC + 6)
-#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_PAYLOAD (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ETHERTYPE + 2)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_FLAGS (ZT_PACKET_IDX_PAYLOAD)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_NETWORK_ID (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_FLAGS + 1)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SUBMITTER_ADDRESS (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_NETWORK_ID + 8)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SOURCE_MAC (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SUBMITTER_ADDRESS + 5)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_DESTINATION_MAC (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SOURCE_MAC + 6)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ADI (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_DESTINATION_MAC + 6)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_BLOOM_FILTER (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ADI + 4)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_HOP_COUNT (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_BLOOM_FILTER + 64)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ETHERTYPE (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_HOP_COUNT + 1)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_PAYLOAD_LENGTH (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_ETHERTYPE + 2)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SIGNATURE_LENGTH (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_PAYLOAD_LENGTH + 2)
+#define ZT_PROTO_VERB_MULTICAST_FRAME_IDX_PAYLOAD (ZT_PROTO_VERB_MULTICAST_FRAME_IDX_SIGNATURE_LENGTH + 2)
 
 // Field indices for parsing OK and ERROR payloads of replies
 #define ZT_PROTO_VERB_HELLO__OK__IDX_TIMESTAMP (ZT_PROTO_VERB_OK_IDX_PAYLOAD)
@@ -413,20 +423,8 @@ public:
 		 */
 		VERB_FRAME = 6,
 
-		/* A multicast frame:
-		 *   <[8] 64-bit network ID>
-		 *   <[6] destination multicast Ethernet address>
-		 *   <[4] multicast additional distinguishing information (ADI)>
-		 *   <[32] multicast propagation bloom filter>
-		 *   <[1] 8-bit strict propagation hop count>
-		 *   <[2] 16-bit average peer multicast bandwidth load>
-		 *   <[6] source Ethernet address>
-		 *   <[2] 16-bit ethertype>
-		 *   <[...] ethernet payload>
-		 *
-		 * No OK or ERROR is generated.
-		 */
-		VERB_MULTICAST_FRAME = 7,
+		/* 7 - old VERB_MULTICAST_FRAME, might be reused once all old 0.2
+		 * clients are off the net. */
 
 		/* Announce interest in multicast group(s):
 		 *   <[8] 64-bit network ID>
@@ -436,7 +434,36 @@ public:
 		 *
 		 * OK is generated on successful receipt.
 		 */
-		VERB_MULTICAST_LIKE = 8
+		VERB_MULTICAST_LIKE = 8,
+
+		/* A multicast frame:
+		 *   <[1] flags, currently unused and must be 0>
+		 *   <[8] 64-bit network ID>
+		 *   <[5] ZeroTier address of original submitter of this multicast>
+		 *   <[6] source MAC address>
+		 *   <[6] destination multicast Ethernet address>
+		 *   <[4] multicast additional distinguishing information (ADI)>
+		 *   <[64] multicast propagation bloom filter>
+		 *   <[1] 8-bit propagation hop count>
+		 *   <[2] 16-bit ethertype>
+		 *   <[2] 16-bit length of payload>
+		 *   <[2] 16-bit length of signature>
+		 *   <[...] ethernet payload>
+		 *   <[...] ECDSA signature>
+		 *
+		 * The signature is made using the key of the original submitter, and
+		 * can be used to authenticate the submitter for security and rate
+		 * control purposes. Fields in the signature are: network ID, source
+		 * MAC, destination MAC, multicast ADI, ethertype, and payload. All
+		 * integers are hashed in big-endian byte order. A zero byte is added
+		 * to the hash between each field.
+		 *
+		 * In the future flags could indicate additional fields appended to the
+		 * end or a different signature algorithm.
+		 *
+		 * No OK or ERROR is generated.
+		 */
+		VERB_MULTICAST_FRAME = 9
 	};
 
 	/**
@@ -535,6 +562,18 @@ public:
 		setVerb(v);
 	}
 
+	/**
+	 * Generate a new IV / packet ID in place
+	 *
+	 * This can be used to re-use a packet buffer multiple times to send
+	 * technically different but otherwise identical copies of the same
+	 * packet.
+	 */
+	inline void newInitializationVector()
+	{
+		Utils::getSecureRandom(field(ZT_PACKET_IDX_IV,8),8);
+	}
+
 	/**
 	 * Set this packet's destination
 	 * 

node/PacketDecoder.hpp

@@ -0,0 +1,140 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_PACKETDECODER_HPP
+#define _ZT_PACKETDECODER_HPP
+
+#include <stdexcept>
+
+#include "Packet.hpp"
+#include "Demarc.hpp"
+#include "InetAddress.hpp"
+#include "Utils.hpp"
+#include "SharedPtr.hpp"
+#include "AtomicCounter.hpp"
+#include "Peer.hpp"
+
+namespace ZeroTier {
+
+class RuntimeEnvironment;
+
+/**
+ * Subclass of packet that handles the decoding of it
+ */
+class PacketDecoder : public Packet
+{
+	friend class SharedPtr<PacketDecoder>;
+
+public:
+	/**
+	 * Create a new packet-in-decode
+	 *
+	 * @param b Source buffer with raw packet data
+	 * @param localPort Local port on which packet was received
+	 * @param remoteAddress Address from which packet came
+	 * @throws std::out_of_range Range error processing packet
+	 */
+	template<unsigned int C2>
+	PacketDecoder(const Buffer<C2> &b,Demarc::Port localPort,const InetAddress &remoteAddress)
+ 		throw(std::out_of_range) :
+ 		Packet(b),
+ 		_receiveTime(Utils::now()),
+ 		_localPort(localPort),
+ 		_remoteAddress(remoteAddress),
+ 		_step(DECODE_STEP_WAITING_FOR_SENDER_LOOKUP),
+ 		__refCount()
+	{
+	}
+
+	/**
+	 * Attempt to decode this packet
+	 *
+	 * Note that this returns 'true' if processing is complete. This says nothing
+	 * about whether the packet was valid. A rejection is 'complete.'
+	 *
+	 * Once true is returned, this should not be called again.
+	 *
+	 * @param _r Runtime environment
+	 * @return True if decoding and processing is complete, false if caller should try again
+	 * @throws std::out_of_range Range error processing packet (should be discarded)
+	 * @throws std::runtime_error Other error processing packet (should be discarded)
+	 */
+	bool tryDecode(const RuntimeEnvironment *_r)
+		throw(std::out_of_range,std::runtime_error);
+
+	/**
+	 * @return Time of packet receipt
+	 */
+	inline uint64_t receiveTime() const throw() { return _receiveTime; }
+
+private:
+	struct _CBaddPeerFromHello_Data
+	{
+		const RuntimeEnvironment *renv;
+		Address source;
+		InetAddress remoteAddress;
+		int localPort;
+		unsigned int vMajor,vMinor,vRevision;
+		uint64_t helloPacketId;
+		uint64_t helloTimestamp;
+	};
+	static void _CBaddPeerFromHello(
+		void *arg, // _CBaddPeerFromHello_Data
+		const SharedPtr<Peer> &p,
+		Topology::PeerVerifyResult result);
+
+	static void _CBaddPeerFromWhois(
+		void *arg, // RuntimeEnvironment
+		const SharedPtr<Peer> &p,
+		Topology::PeerVerifyResult result);
+
+	// These are called internally to handle packet contents once it has
+	// been authenticated, decrypted, decompressed, and classified.
+	bool _doERROR(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doHELLO(const RuntimeEnvironment *_r);
+	bool _doOK(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doWHOIS(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doRENDEZVOUS(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doFRAME(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doMULTICAST_LIKE(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doMULTICAST_FRAME(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+
+	uint64_t _receiveTime;
+	Demarc::Port _localPort;
+	InetAddress _remoteAddress;
+
+	enum {
+		DECODE_STEP_WAITING_FOR_SENDER_LOOKUP, // on initial receipt, we need peer's identity
+		DECODE_STEP_WAITING_FOR_ORIGINAL_SUBMITTER_LOOKUP // this only applies to MULTICAST_FRAME
+	} _step;
+
+	AtomicCounter __refCount;
+};
+
+} // namespace ZeroTier
+
+#endif

node/Peer.cpp

@@ -30,6 +30,14 @@
 namespace ZeroTier {
 
 Peer::Peer() :
+	_id(),
+	_ipv4p(),
+	_ipv6p(),
+	_lastUnicastFrame(0),
+	_lastMulticastFrame(0),
+	_vMajor(0),
+	_vMinor(0),
+	_vRevision(0),
 	_dirty(false)
 {
 }
@@ -37,37 +45,44 @@ Peer::Peer() :
 Peer::Peer(const Identity &myIdentity,const Identity &peerIdentity)
 	throw(std::runtime_error) :
 	_id(peerIdentity),
+	_ipv4p(),
+	_ipv6p(),
+	_lastUnicastFrame(0),
+	_lastMulticastFrame(0),
+	_vMajor(0),
+	_vMinor(0),
+	_vRevision(0),
 	_dirty(true)
 {
 	if (!myIdentity.agree(peerIdentity,_keys,sizeof(_keys)))
 		throw std::runtime_error("new peer identity key agreement failed");
 }
 
-void Peer::onReceive(const RuntimeEnvironment *_r,Demarc::Port localPort,const InetAddress &fromAddr,unsigned int latency,unsigned int hops,Packet::Verb verb,uint64_t now)
+void Peer::onReceive(const RuntimeEnvironment *_r,Demarc::Port localPort,const InetAddress &remoteAddr,unsigned int hops,Packet::Verb verb,uint64_t now)
 {
 	if (!hops) { // direct packet
-		WanPath *wp = (fromAddr.isV4() ? &_ipv4p : &_ipv6p);
-
+		WanPath *wp = (remoteAddr.isV4() ? &_ipv4p : &_ipv6p);
 		wp->lastReceive = now;
-		if (verb == Packet::VERB_FRAME)
-			wp->lastUnicastFrame = now;
-		if (latency)
-			wp->latency = latency;
 		wp->localPort = localPort;
 		if (!wp->fixed)
-			wp->addr = fromAddr;
+			wp->addr = remoteAddr;
+		_dirty = true;
+	}
 
+	if (verb == Packet::VERB_FRAME) {
+		_lastUnicastFrame = now;
+		_dirty = true;
+	} else if (verb == Packet::VERB_MULTICAST_FRAME) {
+		_lastMulticastFrame = now;
 		_dirty = true;
 	}
 }
 
-bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,bool relay,Packet::Verb verb,uint64_t now)
+bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,uint64_t now)
 {
 	if ((_ipv6p.isActive(now))||((!(_ipv4p.addr))&&(_ipv6p.addr))) {
 		if (_r->demarc->send(_ipv6p.localPort,_ipv6p.addr,data,len,-1)) {
 			_ipv6p.lastSend = now;
-			if (verb == Packet::VERB_FRAME)
-				_ipv6p.lastUnicastFrame = now;
 			_dirty = true;
 			return true;
 		}
@@ -76,8 +91,6 @@ bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,b
 	if (_ipv4p.addr) {
 		if (_r->demarc->send(_ipv4p.localPort,_ipv4p.addr,data,len,-1)) {
 			_ipv4p.lastSend = now;
-			if (verb == Packet::VERB_FRAME)
-				_ipv4p.lastUnicastFrame = now;
 			_dirty = true;
 			return true;
 		}
@@ -86,6 +99,17 @@ bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,b
 	return false;
 }
 
+void Peer::onSent(const RuntimeEnvironment *_r,bool relay,Packet::Verb verb,uint64_t now)
+{
+	if (verb == Packet::VERB_FRAME) {
+		_lastUnicastFrame = now;
+		_dirty = true;
+	} else if (verb == Packet::VERB_MULTICAST_FRAME) {
+		_lastMulticastFrame = now;
+		_dirty = true;
+	}
+}
+
 bool Peer::sendFirewallOpener(const RuntimeEnvironment *_r,uint64_t now)
 {
 	bool sent = false;

node/Peer.hpp

@@ -53,15 +53,16 @@
 #define ZT_PEER_MAX_SERIALIZED_LENGTH ( \
 	64 + \
 	IDENTITY_MAX_BINARY_SERIALIZED_LENGTH + \
-	(( \
-		(sizeof(uint64_t) * 5) + \
+	( ( \
+		(sizeof(uint64_t) * 4) + \
 		sizeof(uint16_t) + \
 		1 + \
 		sizeof(uint16_t) + \
 		16 + \
 		1 \
 	) * 2) + \
-	64 \
+	sizeof(uint64_t) + \
+	sizeof(uint64_t) \
 )
 
 namespace ZeroTier {
@@ -110,33 +111,42 @@ public:
 	/**
 	 * Must be called on authenticated packet receive from this peer
 	 * 
+	 * This must be called only after a packet has passed authentication
+	 * checking. Packets that fail are silently discarded.
+	 *
 	 * @param _r Runtime environment
 	 * @param localPort Local port on which packet was received
-	 * @param fromAddr Internet address of sender
-	 * @param latency Latency or 0 if unknown
+	 * @param remoteAddr Internet address of sender
 	 * @param hops ZeroTier (not IP) hops
 	 * @param verb Packet verb
 	 * @param now Current time
 	 */
-	void onReceive(const RuntimeEnvironment *_r,Demarc::Port localPort,const InetAddress &fromAddr,unsigned int latency,unsigned int hops,Packet::Verb verb,uint64_t now);
+	void onReceive(const RuntimeEnvironment *_r,Demarc::Port localPort,const InetAddress &remoteAddr,unsigned int hops,Packet::Verb verb,uint64_t now);
 
 	/**
-	 * Send a UDP packet to this peer
-	 * 
-	 * If the active link is timed out (no receives for ping timeout ms), then
-	 * the active link number is incremented after send. This causes sends to
-	 * cycle through links if there is no clear active link. This also happens
-	 * if the send fails for some reason.
+	 * Send a packet to this peer
 	 * 
 	 * @param _r Runtime environment
 	 * @param data Data to send
 	 * @param len Length of packet
-	 * @param relay This is a relay on behalf of another peer (verb is ignored)
-	 * @param verb Packet verb (if not relay)
 	 * @param now Current time
 	 * @return True if packet appears to have been sent, false on local failure
 	 */
-	bool send(const RuntimeEnvironment *_r,const void *data,unsigned int len,bool relay,Packet::Verb verb,uint64_t now);
+	bool send(const RuntimeEnvironment *_r,const void *data,unsigned int len,uint64_t now);
+
+	/**
+	 * Must be called after a packet is successfully sent to this peer
+	 *
+	 * Note that 'relay' means we've sent a packet *from* this node to this
+	 * peer by relaying it, not that we have relayed a packet from somewhere
+	 * else to this peer. In the latter case this is not called.
+	 *
+	 * @param _r Runtime environment
+	 * @param relay If true, packet was sent indirectly via a relay
+	 * @param verb Packet verb
+	 * @param now Current time
+	 */
+	void onSent(const RuntimeEnvironment *_r,bool relay,Packet::Verb verb,uint64_t now);
 
 	/**
 	 * Send firewall opener to active link
@@ -195,7 +205,25 @@ public:
 	uint64_t lastUnicastFrame() const
 		throw()
 	{
-		return std::max(_ipv4p.lastUnicastFrame,_ipv6p.lastUnicastFrame);
+		return _lastUnicastFrame;
+	}
+
+	/**
+	 * @return Time of most recent multicast frame
+	 */
+	uint64_t lastMulticastFrame() const
+		throw()
+	{
+		return _lastMulticastFrame;
+	}
+
+	/**
+	 * @return Time of most recent frame of any kind (unicast or multicast)
+	 */
+	uint64_t lastFrame() const
+		throw()
+	{
+		return std::max(_lastUnicastFrame,_lastMulticastFrame);
 	}
 
 	/**
@@ -213,6 +241,21 @@ public:
 		return 0;
 	}
 
+	/**
+	 * @param addr Remote address
+	 * @param latency Latency measurment
+	 */
+	void setLatency(const InetAddress &addr,unsigned int latency)
+	{
+		if (addr == _ipv4p.addr) {
+			_ipv4p.latency = latency;
+			_dirty = true;
+		} else if (addr == _ipv6p.addr) {
+			_ipv6p.latency = latency;
+			_dirty = true;
+		}
+	}
+
 	/**
 	 * @return True if this peer has at least one direct IP address path
 	 */
@@ -290,6 +333,20 @@ public:
 		return (_keys + 32); // mac key is second 32-byte key
 	}
 
+	/**
+	 * Set the remote version of the peer (not persisted)
+	 *
+	 * @param vmaj Major version
+	 * @param vmin Minor version
+	 * @param vrev Revision
+	 */
+	inline void setRemoteVersion(unsigned int vmaj,unsigned int vmin,unsigned int vrev)
+	{
+		_vMajor = vmaj;
+		_vMinor = vmin;
+		_vRevision = vrev;
+	}
+
 	/**
 	 * Get and reset dirty flag
 	 * 
@@ -312,11 +369,13 @@ public:
 	inline void serialize(Buffer<C> &b)
 		throw(std::out_of_range)
 	{
-		b.append((unsigned char)1); // version
+		b.append((unsigned char)2); // version
 		b.append(_keys,sizeof(_keys));
 		_id.serialize(b,false);
 		_ipv4p.serialize(b);
 		_ipv6p.serialize(b);
+		b.append(_lastUnicastFrame);
+		b.append(_lastMulticastFrame);
 	}
 
 	template<unsigned int C>
@@ -325,14 +384,19 @@ public:
 	{
 		unsigned int p = startAt;
 
-		if (b[p++] != 1)
+		if (b[p++] != 2)
 			throw std::invalid_argument("Peer: deserialize(): version mismatch");
 
 		memcpy(_keys,b.field(p,sizeof(_keys)),sizeof(_keys)); p += sizeof(_keys);
 		p += _id.deserialize(b,p);
 		p += _ipv4p.deserialize(b,p);
 		p += _ipv6p.deserialize(b,p);
+		_lastUnicastFrame = b.template at<uint64_t>(p); p += sizeof(uint64_t);
+		_lastMulticastFrame = b.template at<uint64_t>(p); p += sizeof(uint64_t);
 
+		_vMajor = 0;
+		_vMinor = 0;
+		_vRevision = 0;
 		_dirty = false;
 
 		return (p - startAt);
@@ -372,7 +436,6 @@ private:
 		WanPath() :
 			lastSend(0),
 			lastReceive(0),
-			lastUnicastFrame(0),
 			lastFirewallOpener(0),
 			localPort(Demarc::ANY_PORT),
 			latency(0),
@@ -393,7 +456,6 @@ private:
 		{
 			b.append(lastSend);
 			b.append(lastReceive);
-			b.append(lastUnicastFrame);
 			b.append(lastFirewallOpener);
 			b.append(Demarc::portToInt(localPort));
 			b.append((uint16_t)latency);
@@ -423,7 +485,6 @@ private:
 
 			lastSend = b.template at<uint64_t>(p); p += sizeof(uint64_t);
 			lastReceive = b.template at<uint64_t>(p); p += sizeof(uint64_t);
-			lastUnicastFrame = b.template at<uint64_t>(p); p += sizeof(uint64_t);
 			lastFirewallOpener = b.template at<uint64_t>(p); p += sizeof(uint64_t);
 			localPort = Demarc::intToPort(b.template at<uint64_t>(p)); p += sizeof(uint64_t);
 			latency = b.template at<uint16_t>(p); p += sizeof(uint16_t);
@@ -449,9 +510,8 @@ private:
 
 		uint64_t lastSend;
 		uint64_t lastReceive;
-		uint64_t lastUnicastFrame;
 		uint64_t lastFirewallOpener;
-		Demarc::Port localPort; // ANY_PORT if not defined
+		Demarc::Port localPort; // ANY_PORT if not defined (size: uint64_t)
 		unsigned int latency; // 0 if never determined
 		InetAddress addr; // null InetAddress if path is undefined
 		bool fixed; // do not learn address from received packets
@@ -463,8 +523,12 @@ private:
 	WanPath _ipv4p;
 	WanPath _ipv6p;
 
+	uint64_t _lastUnicastFrame;
+	uint64_t _lastMulticastFrame;
+
 	// Fields below this line are not persisted with serialize()
 
+	unsigned int _vMajor,_vMinor,_vRevision;
 	bool _dirty;
 
 	AtomicCounter __refCount;
@@ -472,4 +536,13 @@ private:
 
 } // namespace ZeroTier
 
+// Add a swap() for shared ptr's to peers to speed up peer sorts
+namespace std {
+	template<>
+	inline void swap(ZeroTier::SharedPtr<ZeroTier::Peer> &a,ZeroTier::SharedPtr<ZeroTier::Peer> &b)
+	{
+		a.swap(b);
+	}
+}
+
 #endif

node/RuntimeEnvironment.hpp

@@ -30,6 +30,7 @@
 
 #include <string>
 #include "Identity.hpp"
+#include "Condition.hpp"
 
 namespace ZeroTier {
 
@@ -39,6 +40,8 @@ class Demarc;
 class Switch;
 class Topology;
 class SysEnv;
+class Multicaster;
+class CMWC4096;
 
 /**
  * Holds global state for an instance of ZeroTier::Node
@@ -58,8 +61,10 @@ public:
 	RuntimeEnvironment() :
 		identity(),
 		log((Logger *)0),
+		prng((CMWC4096 *)0),
 		nc((NodeConfig *)0),
 		demarc((Demarc *)0),
+		multicaster((Multicaster *)0),
 		sw((Switch *)0),
 		topology((Topology *)0)
 	{
@@ -71,12 +76,17 @@ public:
 	std::string ownershipVerificationSecret;
 	std::string ownershipVerificationSecretHash; // base64 of SHA-256 X16 rounds
 
+	// signal() to prematurely interrupt main loop wait
+	Condition mainLoopWaitCondition;
+
 	Identity configAuthority;
 	Identity identity;
 
 	Logger *log; // may be null
+	CMWC4096 *prng;
 	NodeConfig *nc;
 	Demarc *demarc;
+	Multicaster *multicaster;
 	Switch *sw;
 	Topology *topology;
 	SysEnv *sysEnv;

node/SharedPtr.hpp

@@ -88,6 +88,14 @@ public:
 		return *this;
 	}
 
+	inline void swap(SharedPtr &with)
+		throw()
+	{
+		T *tmp = _ptr;
+		_ptr = with._ptr;
+		with._ptr = tmp;
+	}
+
 	inline operator bool() const throw() { return (_ptr); }
 	inline T &operator*() const throw() { return *_ptr; }
 	inline T *operator->() const throw() { return _ptr; }

node/Switch.hpp

@@ -31,6 +31,7 @@
 #include <map>
 #include <set>
 #include <vector>
+#include <list>
 
 #include "Mutex.hpp"
 #include "MAC.hpp"
@@ -44,6 +45,8 @@
 #include "Network.hpp"
 #include "SharedPtr.hpp"
 #include "Demarc.hpp"
+#include "Multicaster.hpp"
+#include "PacketDecoder.hpp"
 
 namespace ZeroTier {
 
@@ -110,11 +113,11 @@ public:
 	 * Send a HELLO announcement immediately to the indicated address
 	 *
 	 * @param localPort Originating local port or ANY_PORT to pick
-	 * @param addr IP address to send to
+	 * @param remoteAddr IP address to send to
 	 * @param dest Destination peer
 	 * @return True if send appears successful
 	 */
-	bool sendHELLO(const SharedPtr<Peer> &dest,Demarc::Port localPort,const InetAddress &addr);
+	bool sendHELLO(const SharedPtr<Peer> &dest,Demarc::Port localPort,const InetAddress &remoteAddr);
 
 	/**
 	 * Send RENDEZVOUS to two peers to permit them to directly connect
@@ -132,6 +135,14 @@ public:
 	 */
 	bool unite(const Address &p1,const Address &p2,bool force);
 
+	/**
+	 * Send NAT traversal messages to peer at the given candidate address
+	 *
+	 * @param peer Peer to contact
+	 * @param atAddr Address of peer
+	 */
+	void contact(const SharedPtr<Peer> &peer,const InetAddress &atAddr);
+
 	/**
 	 * Perform retries and other periodic timer tasks
 	 * 
@@ -149,70 +160,44 @@ public:
 	 */
 	void announceMulticastGroups(const std::map< SharedPtr<Network>,std::set<MulticastGroup> > &allMemberships);
 
+	/**
+	 * Request WHOIS on a given address
+	 *
+	 * @param addr Address to look up
+	 */
+	void requestWhois(const Address &addr);
+
+	/**
+	 * Run any processes that are waiting for this peer
+	 *
+	 * Called when we learn of a peer's identity from HELLO, OK(WHOIS), etc.
+	 *
+	 * @param peer New peer
+	 */
+	void doAnythingWaitingForPeer(const SharedPtr<Peer> &peer);
+
 private:
-	// Returned by _send() and _processRemotePacket() to indicate what happened
-	enum PacketServiceAttemptResult
-	{
-		PACKET_SERVICE_ATTEMPT_OK,
-		PACKET_SERVICE_ATTEMPT_PEER_UNKNOWN,
-		PACKET_SERVICE_ATTEMPT_SEND_FAILED
-	};
+	void _handleRemotePacketFragment(
+		Demarc::Port localPort,
+		const InetAddress &fromAddr,
+		const Buffer<4096> &data);
 
-	struct _CBaddPeerFromHello_Data
-	{
-		Switch *parent;
-		Address source;
-		InetAddress fromAddr;
-		int localPort;
-		unsigned int vMajor,vMinor,vRevision;
-		uint64_t helloPacketId;
-		uint64_t helloTimestamp;
-	};
-	static void _CBaddPeerFromHello(void *arg,const SharedPtr<Peer> &p,Topology::PeerVerifyResult result);
-	static void _CBaddPeerFromWhois(void *arg,const SharedPtr<Peer> &p,Topology::PeerVerifyResult result); // arg == this
+	void _handleRemotePacketHead(
+		Demarc::Port localPort,
+		const InetAddress &fromAddr,
+		const Buffer<4096> &data);
 
-	void _propagateMulticast(const SharedPtr<Network> &network,const Address &upstream,unsigned char *bloom,const MulticastGroup &mg,unsigned int mcHops,unsigned int mcLoadFactor,const MAC &from,unsigned int etherType,const void *data,unsigned int len);
-	PacketServiceAttemptResult _tryHandleRemotePacket(Demarc::Port localPort,const InetAddress &fromAddr,Packet &packet);
-	void _doHELLO(Demarc::Port localPort,const InetAddress &fromAddr,Packet &packet);
-	void _requestWhois(const Address &addr);
-	Address _sendWhoisRequest(const Address &addr,const Address *peersAlreadyConsulted,unsigned int numPeersAlreadyConsulted);
-	PacketServiceAttemptResult _trySend(const Packet &packet,bool encrypt);
-	void _retryPendingFor(const Address &addr);
+	Address _sendWhoisRequest(
+		const Address &addr,
+		const Address *peersAlreadyConsulted,
+		unsigned int numPeersAlreadyConsulted);
 
-	// Updates entry for crc in multicast history, returns true if already
-	// present in history and not expired.
-	inline bool _checkAndUpdateMulticastHistory(const MAC &fromMac,const MAC &toMulticastMac,const void *payload,unsigned int len,const uint64_t nwid,const uint64_t now)
-	{
-		uint64_t crc = Utils::crc64(0,fromMac.data,6);
-		crc = Utils::crc64(crc,toMulticastMac.data,6);
-		crc = Utils::crc64(crc,payload,len);
-		crc += nwid; // also include network ID
-
-		uint64_t earliest = 0xffffffffffffffffULL;
-		unsigned long earliestIdx = 0;
-		for(unsigned int i=0;i<ZT_MULTICAST_DEDUP_HISTORY_LENGTH;++i) {
-			if (_multicastHistory[i][0] == crc) {
-				uint64_t then = _multicastHistory[i][1];
-				_multicastHistory[i][1] = now;
-				return ((now - then) < ZT_MULTICAST_DEDUP_HISTORY_EXPIRE);
-			} else if (_multicastHistory[i][1] < earliest) {
-				earliest = _multicastHistory[i][1];
-				earliestIdx = i;
-			}
-		}
-
-		_multicastHistory[earliestIdx][0] = crc; // replace oldest entry
-		_multicastHistory[earliestIdx][1] = now;
-
-		return false;
-	}
+	bool _trySend(
+		const Packet &packet,
+		bool encrypt);
 
 	const RuntimeEnvironment *const _r;
 
-	// Multicast packet CRC64's for packets we've received recently, to reject
-	// duplicates during propagation. [0] is CRC64, [1] is time.
-	uint64_t _multicastHistory[ZT_MULTICAST_DEDUP_HISTORY_LENGTH][2];
-
 	struct WhoisRequest
 	{
 		uint64_t lastSent;
@@ -222,29 +207,28 @@ private:
 	std::map< Address,WhoisRequest > _outstandingWhoisRequests;
 	Mutex _outstandingWhoisRequests_m;
 
+	std::list< SharedPtr<PacketDecoder> > _rxQueue;
+	Mutex _rxQueue_m;
+
 	struct TXQueueEntry
 	{
+		TXQueueEntry() {}
+		TXQueueEntry(uint64_t ct,const Packet &p,bool enc) :
+			creationTime(ct),
+			packet(p),
+			encrypt(enc) {}
+
 		uint64_t creationTime;
 		Packet packet; // unencrypted/untagged for TX queue
 		bool encrypt;
 	};
-	std::multimap< Address,TXQueueEntry > _txQueue; // by destination address
+	std::multimap< Address,TXQueueEntry > _txQueue;
 	Mutex _txQueue_m;
 
-	struct RXQueueEntry
-	{
-		uint64_t creationTime;
-		Demarc::Port localPort;
-		Packet packet; // encrypted/tagged
-		InetAddress fromAddr;
-	};
-	std::multimap< Address,RXQueueEntry > _rxQueue; // by source address
-	Mutex _rxQueue_m;
-
 	struct DefragQueueEntry
 	{
 		uint64_t creationTime;
-		Packet frag0;
+		SharedPtr<PacketDecoder> frag0;
 		Packet::Fragment frags[ZT_MAX_PACKET_FRAGMENTS - 1];
 		unsigned int totalFragments; // 0 if only frag0 received, waiting for frags
 		uint32_t haveFragments; // bit mask, LSB to MSB
@@ -255,14 +239,22 @@ private:
 	std::map< Array< Address,2 >,uint64_t > _lastUniteAttempt; // key is always sorted in ascending order, for set-like behavior
 	Mutex _lastUniteAttempt_m;
 
-	struct RendezvousQueueEntry
+	struct ContactQueueEntry
 	{
-		InetAddress inaddr;
+		ContactQueueEntry() {}
+		ContactQueueEntry(const SharedPtr<Peer> &p,uint64_t ft,Demarc::Port lp,const InetAddress &a) :
+			peer(p),
+			fireAtTime(ft),
+			localPort(lp),
+			inaddr(a) {}
+
+		SharedPtr<Peer> peer;
 		uint64_t fireAtTime;
 		Demarc::Port localPort;
+		InetAddress inaddr;
 	};
-	std::map< Address,RendezvousQueueEntry > _rendezvousQueue;
-	Mutex _rendezvousQueue_m;
+	std::list<ContactQueueEntry> _contactQueue;
+	Mutex _contactQueue_m;
 };
 
 } // namespace ZeroTier