0.3.0: BREAKS PROTOCOL BACKWARD COMPATIBILITY

This version is not compatible with versions prior to 0.3.0, so
'git pull' and restart if you are following the alpha.

Changes from 0.2.5:

 - All multicast frames are now signed by the original sender. This
   will permit very efficient and fault tolerant rate limitation
   across networks, and imposes a kind of "hash cash" cost on those
   who wish to flood the network by forcing them to keep regenerating
   new identities.

 - Simplified peer last unicast / last multicast accounting.

 - Improvements to multicast propagation algorithm to achieve better
   coverage with less redundant messages.

 - The bloated Switch class went on a diet, having packet decoding
   broken out into PacketDecoder and multicast propagation algorithm
   broken out into Multicaster.

 - Multicaster is implemented as a template mockable class to permit
   future simulations of huge scale multicast using the actual code
   instead of mockups in another language.

 - Introduced a faster non-cryptographic random source for things
   like multicast propagation and address choosing.

 - Some code cleanup, removal of outdated comments, etc.

Makefile.linux

@@ -6,7 +6,7 @@ ARCH=$(shell uname -m)
 DEFS=-DZT_ARCH="$(ARCH)" -DZT_OSNAME="linux" -DZT_TRACE
 
 # Uncomment for a release optimized build
-CFLAGS=-Wall -O3 -fno-unroll-loops -pthread $(INCLUDES) -DNDEBUG $(DEFS)
+CFLAGS=-Wall -O3 -fno-unroll-loops -fstack-protector -pthread $(INCLUDES) -DNDEBUG $(DEFS)
 STRIP=strip --strip-all
 
 # Uncomment for a debug build

Makefile.mac

@@ -5,7 +5,7 @@ INCLUDES=-Iext/bin/libcrypto/include -Iext/jsoncpp/include
 DEFS=-DZT_ARCH="x86_combined" -DZT_OSNAME="mac" -DZT_TRACE
 
 # Uncomment for a release optimized universal binary build
-CFLAGS=-arch i386 -arch x86_64 -Wall -O3 -ftree-vectorize -pthread -mmacosx-version-min=10.6 -DNDEBUG $(INCLUDES) $(DEFS)
+CFLAGS=-arch i386 -arch x86_64 -Wall -O3 -ftree-vectorize -fstack-protector -pthread -mmacosx-version-min=10.6 -DNDEBUG $(INCLUDES) $(DEFS)
 STRIP=strip
 
 # Uncomment for a debug build

README.md

@@ -50,9 +50,6 @@ Check out the [blog](http://blog.zerotier.com/) for announcements, in-depth arti
 **Q:** What about privacy? Does this hide my location on the network?  
 **A:** ZeroTier is not a connection anonymizer. Other than encryption, it doesn't do anything special to hide your identity or network location. If you want strong privacy protection there are already very advanced tools like [Tor](https://www.torproject.org) for that, and this isn't trying to duplicate their functionality. At the same time, ZT does not do anything special to harm your privacy either. It's not spyware or snoop-ware.
 
-**Q:** You say "almost unlimited size." Isn't multicast and broadcast traffic eventually going to be too much? What happens then?  
-**A:** ZeroTier One uses an algorithm called *implicit social switching*. The overall maximum number of recipients for a multicast is limited, so if there are too many listeners to a given multicast address then obviously not everyone will receive every message. So who does? Social switching causes multicasts to propagate in the direction of peers to whom you have recently communicated. As a result, multicasts tend to propagate along lines of association. The people most likely to get your service announcements are those with whom you frequently connect.
-
 **Q:** Is this designed to replace IP, BGP, IPv6, routers, etc.?  
 **A:** No. Its purpose is to act as a collaboration tool, a VPN alternative, a network mobility tool, a testbed for the development of software that takes advantage of fully open networking, a virtual LAN party for gamers, and so on, but it's not intended (or able) to replace the core of the Internet.
 
@@ -92,6 +89,7 @@ Check out the [blog](http://blog.zerotier.com/) for announcements, in-depth arti
 
 ----
 
-<a href="http://flattr.com/thing/1611614/ZeroTier-Networks" target="_blank"><img src="http://api.flattr.com/button/flattr-badge-large.png" alt="Flattr this" title="Flattr this" border="0" /></a>
+<a href="http://flattr.com/thing/1611614/ZeroTier-Networks" target="_blank"><img src="http://api.flattr.com/button/flattr-badge-large.png" alt="Flattr this" title="Flattr this" border="0" /></a>  
+[![githalytics.com alpha](https://cruel-carlota.pagodabox.com/59b2cbb9c154bf84bddb4b714402e548 "githalytics.com")](http://githalytics.com/zerotier/ZeroTierOne)
 
 (c)2012-2013 [ZeroTier Networks LLC](https://www.zerotier.com/)

node/BloomFilter.hpp

@@ -0,0 +1,133 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_BLOOMFILTER_HPP
+#define _ZT_BLOOMFILTER_HPP
+
+#include <string.h>
+#include "Utils.hpp"
+
+namespace ZeroTier {
+
+/**
+ * A simple bit field bloom filter
+ *
+ * This actually isn't a total filter, in that it does not contain a hashing
+ * algorithm. It's up to the caller to hash/sum the items being remembered
+ * so that the distribution of 'n' is random.
+ *
+ * @tparam B Size in BITS, must be a multiple of 8
+ */
+template<unsigned int B>
+class BloomFilter
+{
+public:
+	/**
+	 * Construct an empty filter
+	 */
+	BloomFilter()
+		throw()
+	{
+		memset(_field,0,sizeof(_field));
+	}
+
+	/**
+	 * Construct from a raw filter
+	 *
+	 * @param b Raw filter bits, must be exactly bytes() in length, or NULL to construct empty
+	 */
+	BloomFilter(const void *b)
+		throw()
+	{
+		if (b)
+			memcpy(_field,b,sizeof(_field));
+		else memset(_field,0,sizeof(_field));
+	}
+
+	/**
+	 * @return Size of filter in bits
+	 */
+	static inline unsigned int bits() throw() { return B; }
+
+	/**
+	 * @return Size of filter in bytes
+	 */
+	static inline unsigned int bytes() throw() { return (B / 8); }
+
+	/**
+	 * @return Pointer to portable array of bytes of bytes() length representing filter
+	 */
+	inline const unsigned char *data() const throw() { return _field; }
+
+	/**
+	 * Clear all bits in filter
+	 */
+	inline void clear()
+		throw()
+	{
+		memset(_field,0,sizeof(_field));
+	}
+
+	/**
+	 * @param n Value to set
+	 */
+	inline void set(unsigned int n)
+		throw()
+	{
+		n %= B;
+		_field[n / 8] |= (1 << (n % 8));
+	}
+
+	/**
+	 * @param n Value to test
+	 * @return True if value is present is set
+	 */
+	inline bool contains(unsigned int n)
+		throw()
+	{
+		n %= B;
+		return (_field[n / 8] & (1 << (n % 8)));
+	}
+
+	/**
+	 * Clear a random bit in this bloom filter
+	 *
+	 * @param rn Random number
+	 */
+	inline void decay(unsigned int rn)
+		throw()
+	{
+		_field[(rn >> 3) % (B / 8)] &= ~((unsigned char)(1 << (rn & 7)));
+	}
+
+private:
+	unsigned char _field[B / 8];
+};
+
+} // namespace ZeroTier
+
+#endif

node/Buffer.hpp

@@ -240,6 +240,22 @@ public:
 		_l += sizeof(T);
 	}
 
+	/**
+	 * Append a run of bytes
+	 *
+	 * @param c Character value to append
+	 * @param n Number of times to append
+	 * @throws std::out_of_range Attempt to append beyond capacity
+	 */
+	inline void append(unsigned char c,unsigned int n)
+		throw(std::out_of_range)
+	{
+		if ((_l + n) > C)
+			throw std::out_of_range("Buffer: append beyond capacity");
+		for(unsigned int i=0;i<n;++i)
+			_b[_l++] = (char)c;
+	}
+
 	/**
 	 * Append a C-array of bytes
 	 * 
@@ -388,7 +404,7 @@ public:
 		return !(*this < b);
 	}
 
-protected:
+private:
 	unsigned int _l;
 	char ZT_VAR_MAY_ALIAS _b[C];
 };

node/CMWC4096.hpp

@@ -0,0 +1,91 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_CMWC4096_HPP
+#define _ZT_CMWC4096_HPP
+
+#include <stdint.h>
+#include "Utils.hpp"
+
+namespace ZeroTier {
+
+/** 
+ * Complement Multiply With Carry random number generator
+ *
+ * Based on original code posted to Usenet in the public domain by
+ * George Marsaglia. Period is approximately 2^131086.
+ *
+ * This is not used for cryptographic purposes but for a very fast
+ * and high-quality PRNG elsewhere in the code.
+ */
+class CMWC4096
+{
+public:
+	/**
+	 * Construct and initialize from secure random source
+	 */
+	CMWC4096()
+		throw()
+	{
+		Utils::getSecureRandom(Q,sizeof(Q));
+		Utils::getSecureRandom(&c,sizeof(c));
+		c %= 809430660;
+		i = 4095;
+	}
+
+	inline uint32_t next32()
+		throw()
+	{
+		uint32_t __i = ++i & 4095;
+		const uint64_t t = (18782ULL * (uint64_t)Q[__i]) + (uint64_t)c;
+		c = (uint32_t)(t >> 32);
+		uint32_t x = c + (uint32_t)t;
+		const uint32_t p = (uint32_t)(x < c); x += p; c += p;
+		return (Q[__i] = 0xfffffffe - x);
+	}
+
+	inline uint64_t next64()
+		throw()
+	{
+		return ((((uint64_t)next32()) << 32) ^ (uint64_t)next32());
+	}
+
+	inline double nextDouble()
+		throw()
+	{
+		return ((double)(next32()) / 4294967296.0);
+	}
+
+private:
+	uint32_t Q[4096];
+	uint32_t c;
+	uint32_t i;
+};
+
+} // namespace ZeroTier
+
+#endif

node/Constants.hpp

@@ -28,10 +28,55 @@
 #ifndef _ZT_CONSTANTS_HPP
 #define _ZT_CONSTANTS_HPP
 
-// Assume these are little-endian, since we don't support old PPC MACs
-// and all newer Mac or Windows systems are either x86_32, x86_64, or
-// ARM in little-endian mode.
-#if defined(__APPLE__) || defined(_WIN32)
+//
+// This include file also auto-detects and canonicalizes some environment
+// information defines:
+//
+// __LINUX__
+// __APPLE__
+// __UNIX_LIKE__ - any "unix like" OS (BSD, posix, etc.)
+// __WINDOWS__
+//
+// Also makes sure __BYTE_ORDER is defined reasonably.
+//
+
+// Canonicalize Linux... is this necessary? Do it anyway to be defensive.
+#if defined(__linux__) || defined(linux) || defined(__LINUX__) || defined(__linux)
+#ifndef __LINUX__
+#define __LINUX__
+#ifndef __UNIX_LIKE__
+#define __UNIX_LIKE__
+#endif
+#endif
+#endif
+
+// TODO: Android is what? Linux technically, but does it define it?
+
+// OSX and iOS are unix-like OSes far as we're concerned
+#ifdef __APPLE__
+#ifndef __UNIX_LIKE__
+#define __UNIX_LIKE__
+#endif
+#endif
+
+// Linux has endian.h
+#ifdef __LINUX__
+#include <endian.h>
+#endif
+
+#if defined(_WIN32) || defined(_WIN64)
+#ifndef __WINDOWS__
+#define __WINDOWS__
+#endif
+#undef __UNIX_LIKE__
+#define ZT_PATH_SEPARATOR '\\'
+#define ZT_PATH_SEPARATOR_S "\\"
+#define ZT_EOL_S "\r\n"
+#endif
+
+// Assume these are little-endian. PPC is not supported for OSX, and ARM
+// runs in little-endian mode for these OS families.
+#if defined(__APPLE__) || defined(__WINDOWS__)
 #undef __BYTE_ORDER
 #undef __LITTLE_ENDIAN
 #undef __BIG_ENDIAN
@@ -40,33 +85,23 @@
 #define __BYTE_ORDER 1234
 #endif
 
-// Linux has endian.h, which should tell us
-#if defined(__linux__) || defined(linux) || defined(__LINUX__) || defined(__linux)
-#include <endian.h>
-#endif
-
-#ifndef __BYTE_ORDER
-error_no_byte_order_defined
-#endif
-
-#ifndef ZT_OSNAME
-error_no_ZT_OSNAME
-#endif
-
-#ifndef ZT_ARCH
-error_no_ZT_ARCH
-#endif
-
-#ifdef _WIN32
-#define ZT_PATH_SEPARATOR '\\'
-#define ZT_PATH_SEPARATOR_S "\\"
-#define ZT_EOL_S "\r\n"
-#else
+#ifdef __UNIX_LIKE__
 #define ZT_PATH_SEPARATOR '/'
 #define ZT_PATH_SEPARATOR_S "/"
 #define ZT_EOL_S "\n"
 #endif
 
+// Error out if required symbols are missing
+#ifndef __BYTE_ORDER
+error_no_byte_order_defined;
+#endif
+#ifndef ZT_OSNAME
+error_no_ZT_OSNAME_defined;
+#endif
+#ifndef ZT_ARCH
+error_no_ZT_ARCH_defined;
+#endif
+
 /**
  * Length of a ZeroTier address in bytes
  */
@@ -200,12 +235,17 @@ error_no_ZT_ARCH
 /**
  * Length of circular ring buffer history of multicast packets
  */
-#define ZT_MULTICAST_DEDUP_HISTORY_LENGTH 4096
+#define ZT_MULTICAST_DEDUP_HISTORY_LENGTH 1024
 
 /**
  * Expiration time in ms for multicast history items
  */
-#define ZT_MULTICAST_DEDUP_HISTORY_EXPIRE 8000
+#define ZT_MULTICAST_DEDUP_HISTORY_EXPIRE 4000
+
+/**
+ * Number of bits to randomly "decay" in bloom filter per hop
+ */
+#define ZT_MULTICAST_BLOOM_FILTER_DECAY_RATE 2
 
 /**
  * Period between announcements of all multicast 'likes' in ms

node/Demarc.cpp

@@ -143,7 +143,7 @@ Demarc::Port Demarc::pick(const InetAddress &to) const
 			}
 		}
 		if (possibilities.size())
-			return possibilities[Utils::randomInt<unsigned int>() % possibilities.size()]->first;
+			return possibilities[_r->prng->next32() % possibilities.size()]->first;
 		else return NULL_PORT;
 	} catch ( ... ) {
 		return NULL_PORT;
@@ -174,7 +174,7 @@ Demarc::Port Demarc::send(Demarc::Port fromPort,const InetAddress &to,const void
 				}
 			}
 			if (possibilities.size())
-				pe = possibilities[Utils::randomInt<unsigned int>() % possibilities.size()];
+				pe = possibilities[_r->prng->next32() % possibilities.size()];
 			else {
 				_ports_m.unlock();
 				return NULL_PORT;

node/EthernetTap.hpp

@@ -36,14 +36,13 @@
 #include <set>
 #include <string>
 #include <stdexcept>
-#include "Array.hpp"
-#include "Utils.hpp"
-#include "InetAddress.hpp"
-#include "NonCopyable.hpp"
-#include "MAC.hpp"
 #include "Constants.hpp"
+#include "InetAddress.hpp"
+#include "MAC.hpp"
 #include "Mutex.hpp"
 #include "MulticastGroup.hpp"
+#include "Thread.hpp"
+#include "Buffer.hpp"
 
 namespace ZeroTier {
 
@@ -52,21 +51,35 @@ class RuntimeEnvironment;
 /**
  * System ethernet tap device
  */
-class EthernetTap : NonCopyable
+class EthernetTap : protected Thread
 {
 public:
 	/**
 	 * Construct a new TAP device
 	 *
+	 * Handler arguments: arg,from,to,etherType,data
+	 * 
 	 * @param renv Runtime environment
 	 * @param mac MAC address of device
 	 * @param mtu MTU of device
+	 * @param handler Handler function to be called when data is received from the tap
+	 * @param arg First argument to handler function
 	 * @throws std::runtime_error Unable to allocate device
 	 */
-	EthernetTap(const RuntimeEnvironment *renv,const MAC &mac,unsigned int mtu)
+	EthernetTap(
+		const RuntimeEnvironment *renv,
+		const MAC &mac,
+		unsigned int mtu,
+		void (*handler)(void *,const MAC &,const MAC &,unsigned int,const Buffer<4096> &),
+		void *arg)
 		throw(std::runtime_error);
 
-	~EthernetTap();
+	/**
+	 * Close tap and shut down thread
+	 *
+	 * This may block for a few seconds while thread exits.
+	 */
+	virtual ~EthernetTap();
 
 	/**
 	 * Perform OS dependent actions on network configuration change detection
@@ -137,31 +150,10 @@ public:
 	 */
 	void put(const MAC &from,const MAC &to,unsigned int etherType,const void *data,unsigned int len);
 
-	/**
-	 * Get the next packet from the interface, blocking if none is available.
-	 *
-	 * @param from Filled with MAC address of source (normally our own)
-	 * @param to Filled with MAC address of destination
-	 * @param etherType Filled with Ethernet frame type
-	 * @param buf Buffer to fill (must have room for MTU bytes)
-	 * @return Number of bytes read or 0 if none
-	 */
-	unsigned int get(MAC &from,MAC &to,unsigned int &etherType,void *buf);
-
 	/**
 	 * @return OS-specific device or connection name
 	 */
-	std::string deviceName();
-
-	/**
-	 * @return True if tap is open
-	 */
-	bool open() const;
-
-	/**
-	 * Close this tap, invalidating the object and causing get() to abort
-	 */
-	void close();
+	std::string deviceName() const;
 
 	/**
 	 * Fill or modify a set to contain multicast groups for this device
@@ -177,6 +169,10 @@ public:
 	 */
 	bool updateMulticastGroups(std::set<MulticastGroup> &groups);
 
+protected:
+	virtual void main()
+		throw();
+
 private:
 	const MAC _mac;
 	const unsigned int _mtu;
@@ -186,20 +182,14 @@ private:
 	std::set<InetAddress> _ips;
 	Mutex _ips_m;
 
-#if defined(__APPLE__) || defined(__linux__) || defined(linux) || defined(__LINUX__) || defined(__linux)
+	void (*_handler)(void *,const MAC &,const MAC &,unsigned int,const Buffer<4096> &);
+	void *_arg;
 
+#ifdef __UNIX_LIKE__
 	char _dev[16];
-	unsigned char *_putBuf;
-	unsigned char *_getBuf;
 	int _fd;
-
-	bool _isReading;
-	pthread_t _isReadingThreadId;
-	Mutex _isReading_m;
-
-#elif defined(_WIN32) /* -------------------------------------------------- */
-
-#endif /* ----------------------------------------------------------------- */
+	int _shutdownSignalPipe[2];
+#endif
 };
 
 } // namespace ZeroTier

node/Http.cpp

@@ -173,7 +173,7 @@ void Http::Request::main()
 				addrList->sort();
 				addrList->unique();
 				unsigned int i = 0,k = 0;
-				k = Utils::randomInt<unsigned int>() % addrList->size();
+				k = rand() % addrList->size();
 				std::list<InetAddress>::iterator a(addrList->begin());
 				while (i++ != k) ++a;
 				addr = &(*a);

node/Multicaster.hpp

@@ -0,0 +1,325 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_MULTICASTER_HPP
+#define _ZT_MULTICASTER_HPP
+
+#include <stdint.h>
+#include <string.h>
+
+#include <openssl/sha.h>
+
+#include <utility>
+#include <algorithm>
+#include <map>
+#include <set>
+#include <vector>
+#include <string>
+
+#include "Constants.hpp"
+#include "Buffer.hpp"
+#include "Packet.hpp"
+#include "MulticastGroup.hpp"
+#include "Utils.hpp"
+#include "MAC.hpp"
+#include "Address.hpp"
+#include "SharedPtr.hpp"
+#include "BloomFilter.hpp"
+#include "Identity.hpp"
+#include "CMWC4096.hpp"
+
+// Maximum sample size to pick during choice of multicast propagation peers
+#define ZT_MULTICAST_PICK_MAX_SAMPLE_SIZE (ZT_MULTICAST_PROPAGATION_BREADTH * 8)
+
+namespace ZeroTier {
+
+/**
+ * Multicast propagation engine
+ *
+ * This is written as a generic class so that it can be mocked and tested
+ * in simulation. It also always takes 'now' as an argument, permitting
+ * running in simulated time.
+ */
+class Multicaster
+{
+public:
+	/**
+	 * 256-bit simple bloom filter included with multicast frame packets
+	 */
+	typedef BloomFilter<ZT_PROTO_VERB_MULTICAST_FRAME_BLOOM_FILTER_SIZE_BITS> MulticastBloomFilter;
+
+	Multicaster()
+		throw()
+	{
+		memset(_multicastHistory,0,sizeof(_multicastHistory));
+		_multicastHistoryPtr = 0;
+	}
+
+	/**
+	 * Generate a signature of a multicast packet using an identity
+	 *
+	 * @param id Identity to sign with (must have secret key portion)
+	 * @param nwid Network ID
+	 * @param from MAC address of sender
+	 * @param to Multicast group
+	 * @param etherType 16-bit ethernet type
+	 * @param data Ethernet frame data
+	 * @param len Length of frame
+	 * @return ECDSA signature
+	 */
+	static inline std::string signMulticastPacket(const Identity &id,uint64_t nwid,const MAC &from,const MulticastGroup &to,unsigned int etherType,const void *data,unsigned int len)
+	{
+		unsigned char digest[32];
+		_hashMulticastPacketForSig(nwid,from,to,etherType,data,len,digest);
+		return id.sign(digest);
+	}
+
+	/**
+	 * Verify a signature from a multicast packet
+	 *
+	 * @param id Identity of original signer
+	 * @param nwid Network ID
+	 * @param from MAC address of sender
+	 * @param to Multicast group
+	 * @param etherType 16-bit ethernet type
+	 * @param data Ethernet frame data
+	 * @param len Length of frame
+	 * @param signature ECDSA signature
+	 * @param siglen Length of signature in bytes
+	 * @return ECDSA signature
+	 */
+	static bool verifyMulticastPacket(const Identity &id,uint64_t nwid,const MAC &from,const MulticastGroup &to,unsigned int etherType,const void *data,unsigned int len,const void *signature,unsigned int siglen)
+	{
+		unsigned char digest[32];
+		_hashMulticastPacketForSig(nwid,from,to,etherType,data,len,digest);
+		return id.verifySignature(digest,signature,siglen);
+	}
+
+	/**
+	 * Compute the CRC64 code for multicast deduplication
+	 *
+	 * @param nwid Network ID
+	 * @param from Sender MAC
+	 * @param to Destination multicast group
+	 * @param etherType Ethernet frame type
+	 * @param payload Multicast frame data
+	 * @param len Length of frame
+	 */
+	static inline uint64_t computeMulticastDedupCrc(
+		uint64_t nwid,
+		const MAC &from,
+		const MulticastGroup &to,
+		unsigned int etherType,
+		const void *payload,
+		unsigned int len)
+		throw()
+	{
+		// This CRC is only used locally, so byte order issues and
+		// such don't matter. It can also be changed without protocol
+		// impact.
+		uint64_t crc = Utils::crc64(0,from.data,6);
+		crc = Utils::crc64(crc,to.mac().data,6);
+		crc ^= (uint64_t)to.adi();
+		crc ^= (uint64_t)etherType;
+		crc = Utils::crc64(crc,payload,len);
+		crc ^= nwid; // also include network ID in CRC
+		return crc;
+	}
+
+	/**
+	 * Check multicast history to see if this is a duplicate
+	 *
+	 * @param crc Multicast CRC
+	 * @param now Current time
+	 * @return True if this appears to be a duplicate to within history expiration time
+	 */
+	inline bool checkDuplicate(uint64_t crc,uint64_t now) const
+		throw()
+	{
+		for(unsigned int i=0;i<ZT_MULTICAST_DEDUP_HISTORY_LENGTH;++i) {
+			if ((_multicastHistory[i][0] == crc)&&((now - _multicastHistory[i][1]) < ZT_MULTICAST_DEDUP_HISTORY_EXPIRE))
+				return true;
+		}
+		return false;
+	}
+
+	/**
+	 * Add a multicast CRC to the multicast deduplication history
+	 *
+	 * @param crc Multicast CRC
+	 * @param now Current time
+	 */
+	inline void addToDedupHistory(uint64_t crc,uint64_t now)
+		throw()
+	{
+		unsigned int mhi = ++_multicastHistoryPtr % ZT_MULTICAST_DEDUP_HISTORY_LENGTH;
+		_multicastHistory[mhi][0] = crc;
+		_multicastHistory[mhi][1] = now;
+	}
+
+	/**
+	 * Update the most recent LIKE time for an address in a given multicast group on a given network
+	 *
+	 * @param nwid Network ID
+	 * @param mg Multicast group
+	 * @param addr Address that likes group on given network
+	 * @param now Current timestamp
+	 */
+	inline void likesMulticastGroup(const uint64_t nwid,const MulticastGroup &mg,const Address &addr,const uint64_t now)
+	{
+		Mutex::Lock _l(_multicastMemberships_m);
+		std::vector<MulticastMembership> &memberships = _multicastMemberships[MulticastChannel(nwid,mg)];
+		for(std::vector<MulticastMembership>::iterator mm(memberships.begin());mm!=memberships.end();++mm) {
+			if (mm->first == addr) {
+				mm->second = now;
+				return;
+			}
+		}
+		memberships.push_back(MulticastMembership(addr,now));
+	}
+
+	/**
+	 * Choose peers to send a propagating multicast to
+	 *
+	 * @param topology Topology object or mock thereof
+	 * @param nwid Network ID
+	 * @param mg Multicast group
+	 * @param originalSubmitter Original submitter of multicast message to network
+	 * @param upstream Address from which message originated, or null (0) address if none
+	 * @param bf Bloom filter, updated in place with sums of addresses in chosen peers and/or decay
+	 * @param max Maximum number of peers to pick
+	 * @param peers Array of objects of type P to fill with up to [max] peers
+	 * @param now Current timestamp
+	 * @return Number of peers actually stored in peers array
+	 * @tparam T Type of topology, which is Topology in running code or a mock in simulation
+	 * @tparam P Type of peers, which is SharedPtr<Peer> in running code or a mock in simulation (mock must behave like a pointer type)
+	 */
+	template<typename T,typename P>
+	inline unsigned int pickNextPropagationPeers(
+		CMWC4096 &prng,
+		T &topology,
+		uint64_t nwid,
+		const MulticastGroup &mg,
+		const Address &originalSubmitter,
+		const Address &upstream,
+		MulticastBloomFilter &bf,
+		unsigned int max,
+		P *peers,
+		uint64_t now)
+	{
+		typename std::set< P,_PeerPropagationPrioritySortOrder<P> > toConsider;
+
+		// Pick up to ZT_MULTICAST_PICK_MAX_SAMPLE_SIZE peers that have
+		// subscribed to this channel and that are not in bloom filter.
+		// Pick randomly from subscribers, but place into a set that is
+		// sorted in descending order of time of most recent unicast
+		// frame transfer. (Implicit social ordering.) Also ignore original
+		// submitter and upstream, since we know these have seen this
+		// message.
+		{
+			Mutex::Lock _l(_multicastMemberships_m);
+			std::map< MulticastChannel,std::vector<MulticastMembership> >::iterator mm(_multicastMemberships.find(MulticastChannel(nwid,mg)));
+			if ((mm != _multicastMemberships.end())&&(!mm->second.empty())) {
+				for(unsigned int stries=0;stries<ZT_MULTICAST_PICK_MAX_SAMPLE_SIZE;++stries) {
+					MulticastMembership &m = mm->second[prng.next32() % mm->second.size()];
+					if (((now - m.second) < ZT_MULTICAST_LIKE_EXPIRE)&&(!bf.contains(m.first.sum()))&&(m.first != originalSubmitter)&&(m.first != upstream)) {
+						P peer(topology.getPeer(m.first));
+						if (peer)
+							toConsider.insert(peer);
+					}
+				}
+			}
+		}
+
+		// The first peers in toConsider will be the 'best'
+		unsigned int chosen = 0;
+		for(typename std::set< P,_PeerPropagationPrioritySortOrder<P> >::iterator i(toConsider.begin());((i!=toConsider.end())&&(chosen < max));++i)
+			bf.set((peers[chosen++] = *i)->address().sum());
+
+		// Add a supernode if there are fewer than the desired
+		// number of recipients.
+		if (chosen < max) {
+			P peer = topology.getBestSupernode(&originalSubmitter,1,true);
+			if (peer)
+				peers[chosen++] = peer;
+		}
+
+		return chosen;
+	}
+
+private:
+	// Sort order for chosen propagation peers
+	template<typename P>
+	struct _PeerPropagationPrioritySortOrder
+	{
+		inline bool operator()(const P &p1,const P &p2) const
+		{
+			return (p1->lastUnicastFrame() > p2->lastUnicastFrame());
+		}
+	};
+
+	static inline void _hashMulticastPacketForSig(uint64_t nwid,const MAC &from,const MulticastGroup &to,unsigned int etherType,const void *data,unsigned int len,unsigned char *digest)
+		throw()
+	{
+		unsigned char zero = 0;
+		SHA256_CTX sha;
+		SHA256_Init(&sha);
+		uint64_t _nwid = Utils::hton(nwid);
+		SHA256_Update(&sha,(unsigned char *)&_nwid,sizeof(_nwid));
+		SHA256_Update(&sha,&zero,1);
+		SHA256_Update(&sha,(unsigned char *)from.data,6);
+		SHA256_Update(&sha,&zero,1);
+		SHA256_Update(&sha,(unsigned char *)to.mac().data,6);
+		SHA256_Update(&sha,&zero,1);
+		uint32_t _adi = Utils::hton(to.adi());
+		SHA256_Update(&sha,(unsigned char *)&_adi,sizeof(_adi));
+		SHA256_Update(&sha,&zero,1);
+		uint16_t _etype = Utils::hton((uint16_t)etherType);
+		SHA256_Update(&sha,(unsigned char *)&_etype,sizeof(_etype));
+		SHA256_Update(&sha,&zero,1);
+		SHA256_Update(&sha,(unsigned char *)data,len);
+		SHA256_Final(digest,&sha);
+	}
+
+	// ring buffer: [0] - CRC, [1] - timestamp
+	uint64_t _multicastHistory[ZT_MULTICAST_DEDUP_HISTORY_LENGTH][2];
+	volatile unsigned int _multicastHistoryPtr;
+
+	// A multicast channel, essentially a pub/sub channel. It consists of a
+	// network ID and a multicast group within that network.
+	typedef std::pair<uint64_t,MulticastGroup> MulticastChannel;
+
+	// Address and time of last LIKE
+	typedef std::pair<Address,uint64_t> MulticastMembership;
+
+	std::map< MulticastChannel,std::vector<MulticastMembership> > _multicastMemberships;
+	Mutex _multicastMemberships_m;
+};
+
+} // namespace ZeroTier
+
+#endif

node/Network.cpp

@@ -32,49 +32,29 @@ namespace ZeroTier {
 
 Network::Network(const RuntimeEnvironment *renv,uint64_t id)
 	throw(std::runtime_error) :
-	Thread(),
 	_r(renv),
 	_id(id),
-	_tap(renv,renv->identity.address().toMAC(),ZT_IF_MTU),
+	_tap(renv,renv->identity.address().toMAC(),ZT_IF_MTU,&_CBhandleTapData,this),
 	_members(),
 	_open(false),
 	_lock()
 {
-	TRACE("new network %llu created, TAP device: %s",id,_tap.deviceName().c_str());
-	start();
 }
 
 Network::~Network()
 {
-	_tap.close();
-	join();
-	TRACE("network %llu (%s) closed",_id,_tap.deviceName().c_str());
 }
 
-void Network::main()
-	throw()
+void Network::_CBhandleTapData(void *arg,const MAC &from,const MAC &to,unsigned int etherType,const Buffer<4096> &data)
 {
-	Buffer<4096> buf;
-	MAC from,to;
-	unsigned int etherType = 0;
-
-	while (_tap.open()) {
-		unsigned int len = _tap.get(from,to,etherType,buf.data());
-		if (len) {
-			buf.setSize(len);
-			try {
-				if (!*__refCount)
-					break; // sanity check
-				_r->sw->onLocalEthernet(SharedPtr<Network>(this),from,to,etherType,buf);
-			} catch (std::exception &exc) {
-				TRACE("unexpected exception handling local packet: %s",exc.what());
-			} catch ( ... ) {
-				TRACE("unexpected exception handling local packet");
-			}
-		} else break;
+	const RuntimeEnvironment *_r = ((Network *)arg)->_r;
+	try {
+		_r->sw->onLocalEthernet(SharedPtr<Network>((Network *)arg),from,to,etherType,data);
+	} catch (std::exception &exc) {
+		TRACE("unexpected exception handling local packet: %s",exc.what());
+	} catch ( ... ) {
+		TRACE("unexpected exception handling local packet");
 	}
-
-	TRACE("network %llu thread terminating",_id);
 }
 
 } // namespace ZeroTier

node/Network.hpp

@@ -40,8 +40,9 @@
 #include "SharedPtr.hpp"
 #include "AtomicCounter.hpp"
 #include "RuntimeEnvironment.hpp"
-#include "Thread.hpp"
 #include "MulticastGroup.hpp"
+#include "NonCopyable.hpp"
+#include "MAC.hpp"
 
 namespace ZeroTier {
 
@@ -50,17 +51,17 @@ class NodeConfig;
 /**
  * Local network endpoint
  */
-class Network : protected Thread
+class Network : NonCopyable
 {
 	friend class SharedPtr<Network>;
 	friend class NodeConfig;
 
 private:
-	virtual ~Network();
-
 	Network(const RuntimeEnvironment *renv,uint64_t id)
 		throw(std::runtime_error);
 
+	~Network();
+
 public:
 	/**
 	 * @return Network ID
@@ -141,11 +142,9 @@ public:
 		return _multicastGroups;
 	}
 
-protected:
-	virtual void main()
-		throw();
-
 private:
+	static void _CBhandleTapData(void *arg,const MAC &from,const MAC &to,unsigned int etherType,const Buffer<4096> &data);
+
 	const RuntimeEnvironment *_r;
 	uint64_t _id;
 	EthernetTap _tap;

node/Node.cpp

@@ -64,6 +64,8 @@
 #include "Network.hpp"
 #include "MulticastGroup.hpp"
 #include "Mutex.hpp"
+#include "Multicaster.hpp"
+#include "CMWC4096.hpp"
 
 #include "../version.h"
 
@@ -116,8 +118,10 @@ Node::~Node()
 	delete impl->renv.sysEnv;
 	delete impl->renv.topology;
 	delete impl->renv.sw;
+	delete impl->renv.multicaster;
 	delete impl->renv.demarc;
 	delete impl->renv.nc;
+	delete impl->renv.prng;
 	delete impl->renv.log;
 
 	delete impl;
@@ -151,6 +155,8 @@ Node::ReasonForTermination Node::run()
 
 		TRACE("initializing...");
 
+		_r->prng = new CMWC4096();
+
 		if (!_r->configAuthority.fromString(_r->configAuthorityIdentityStr))
 			return impl->terminateBecause(Node::NODE_UNRECOVERABLE_ERROR,"configuration authority identity is not valid");
 
@@ -191,8 +197,11 @@ Node::ReasonForTermination Node::run()
 		std::string ovsPath(_r->homePath + ZT_PATH_SEPARATOR_S + "thisdeviceismine");
 		if (((Utils::now() - Utils::getLastModified(ovsPath.c_str())) >= ZT_OVS_GENERATE_NEW_IF_OLDER_THAN)||(!Utils::readFile(ovsPath.c_str(),_r->ownershipVerificationSecret))) {
 			_r->ownershipVerificationSecret = "";
-			for(unsigned int i=0;i<24;++i)
-				_r->ownershipVerificationSecret.push_back("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"[Utils::randomInt<unsigned int>() % 62]);
+			unsigned int securern = 0;
+			for(unsigned int i=0;i<24;++i) {
+				Utils::getSecureRandom(&securern,sizeof(securern));
+				_r->ownershipVerificationSecret.push_back("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"[securern % 62]);
+			}
 			_r->ownershipVerificationSecret.append(ZT_EOL_S);
 			if (!Utils::writeFile(ovsPath.c_str(),_r->ownershipVerificationSecret))
 				return impl->terminateBecause(Node::NODE_UNRECOVERABLE_ERROR,"could not write 'thisdeviceismine' (home path not writable?)");
@@ -211,6 +220,7 @@ Node::ReasonForTermination Node::run()
 		// watcher.
 		_r->nc = new NodeConfig(_r,_r->autoconfUrlPrefix + _r->identity.address().toString());
 		_r->demarc = new Demarc(_r);
+		_r->multicaster = new Multicaster();
 		_r->sw = new Switch(_r);
 		_r->topology = new Topology(_r,(_r->homePath + ZT_PATH_SEPARATOR_S + "peer.db").c_str());
 		_r->sysEnv = new SysEnv(_r);
@@ -396,7 +406,7 @@ Node::ReasonForTermination Node::run()
 			try {
 				unsigned long delay = std::min((unsigned long)ZT_MIN_SERVICE_LOOP_INTERVAL,_r->sw->doTimerTasks());
 				uint64_t start = Utils::now();
-				Thread::sleep(delay);
+				_r->mainLoopWaitCondition.wait(delay);
 				lastDelayDelta = (long)(Utils::now() - start) - (long)delay;
 			} catch (std::exception &exc) {
 				LOG("unexpected exception running Switch doTimerTasks: %s",exc.what());
@@ -423,12 +433,14 @@ void Node::terminate()
 	throw()
 {
 	((_NodeImpl *)_impl)->terminateNow = true;
+	((_NodeImpl *)_impl)->renv.mainLoopWaitCondition.signal();
 }
 
 void Node::updateStatusNow()
 	throw()
 {
 	((_NodeImpl *)_impl)->updateStatusNow = true;
+	((_NodeImpl *)_impl)->renv.mainLoopWaitCondition.signal();
 }
 
 class _VersionStringMaker

node/NodeConfig.cpp

@@ -152,7 +152,7 @@ void NodeConfig::__CBautoconfHandler(const std::string &lastModified,const std::
 									if (rawAddr.length() == ZT_ADDRESS_LENGTH) {
 										Address addr(rawAddr.data());
 										if ((addr)&&(!addr.isReserved())) {
-											TRACE("network %llu member: %s",nwid,addr.toString().c_str());
+											//TRACE("network %llu member: %s",nwid,addr.toString().c_str());
 											nw->_members.insert(addr);
 										}
 									}

node/PacketDecoder.hpp

@@ -0,0 +1,140 @@
+/*
+ * ZeroTier One - Global Peer to Peer Ethernet
+ * Copyright (C) 2012-2013  ZeroTier Networks LLC
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * --
+ *
+ * ZeroTier may be used and distributed under the terms of the GPLv3, which
+ * are available at: http://www.gnu.org/licenses/gpl-3.0.html
+ *
+ * If you would like to embed ZeroTier into a commercial application or
+ * redistribute it in a modified binary form, please contact ZeroTier Networks
+ * LLC. Start here: http://www.zerotier.com/
+ */
+
+#ifndef _ZT_PACKETDECODER_HPP
+#define _ZT_PACKETDECODER_HPP
+
+#include <stdexcept>
+
+#include "Packet.hpp"
+#include "Demarc.hpp"
+#include "InetAddress.hpp"
+#include "Utils.hpp"
+#include "SharedPtr.hpp"
+#include "AtomicCounter.hpp"
+#include "Peer.hpp"
+
+namespace ZeroTier {
+
+class RuntimeEnvironment;
+
+/**
+ * Subclass of packet that handles the decoding of it
+ */
+class PacketDecoder : public Packet
+{
+	friend class SharedPtr<PacketDecoder>;
+
+public:
+	/**
+	 * Create a new packet-in-decode
+	 *
+	 * @param b Source buffer with raw packet data
+	 * @param localPort Local port on which packet was received
+	 * @param remoteAddress Address from which packet came
+	 * @throws std::out_of_range Range error processing packet
+	 */
+	template<unsigned int C2>
+	PacketDecoder(const Buffer<C2> &b,Demarc::Port localPort,const InetAddress &remoteAddress)
+ 		throw(std::out_of_range) :
+ 		Packet(b),
+ 		_receiveTime(Utils::now()),
+ 		_localPort(localPort),
+ 		_remoteAddress(remoteAddress),
+ 		_step(DECODE_STEP_WAITING_FOR_SENDER_LOOKUP),
+ 		__refCount()
+	{
+	}
+
+	/**
+	 * Attempt to decode this packet
+	 *
+	 * Note that this returns 'true' if processing is complete. This says nothing
+	 * about whether the packet was valid. A rejection is 'complete.'
+	 *
+	 * Once true is returned, this should not be called again.
+	 *
+	 * @param _r Runtime environment
+	 * @return True if decoding and processing is complete, false if caller should try again
+	 * @throws std::out_of_range Range error processing packet (should be discarded)
+	 * @throws std::runtime_error Other error processing packet (should be discarded)
+	 */
+	bool tryDecode(const RuntimeEnvironment *_r)
+		throw(std::out_of_range,std::runtime_error);
+
+	/**
+	 * @return Time of packet receipt
+	 */
+	inline uint64_t receiveTime() const throw() { return _receiveTime; }
+
+private:
+	struct _CBaddPeerFromHello_Data
+	{
+		const RuntimeEnvironment *renv;
+		Address source;
+		InetAddress remoteAddress;
+		int localPort;
+		unsigned int vMajor,vMinor,vRevision;
+		uint64_t helloPacketId;
+		uint64_t helloTimestamp;
+	};
+	static void _CBaddPeerFromHello(
+		void *arg, // _CBaddPeerFromHello_Data
+		const SharedPtr<Peer> &p,
+		Topology::PeerVerifyResult result);
+
+	static void _CBaddPeerFromWhois(
+		void *arg, // RuntimeEnvironment
+		const SharedPtr<Peer> &p,
+		Topology::PeerVerifyResult result);
+
+	// These are called internally to handle packet contents once it has
+	// been authenticated, decrypted, decompressed, and classified.
+	bool _doERROR(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doHELLO(const RuntimeEnvironment *_r);
+	bool _doOK(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doWHOIS(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doRENDEZVOUS(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doFRAME(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doMULTICAST_LIKE(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+	bool _doMULTICAST_FRAME(const RuntimeEnvironment *_r,const SharedPtr<Peer> &peer);
+
+	uint64_t _receiveTime;
+	Demarc::Port _localPort;
+	InetAddress _remoteAddress;
+
+	enum {
+		DECODE_STEP_WAITING_FOR_SENDER_LOOKUP, // on initial receipt, we need peer's identity
+		DECODE_STEP_WAITING_FOR_ORIGINAL_SUBMITTER_LOOKUP // this only applies to MULTICAST_FRAME
+	} _step;
+
+	AtomicCounter __refCount;
+};
+
+} // namespace ZeroTier
+
+#endif

node/Peer.cpp

@@ -30,6 +30,14 @@
 namespace ZeroTier {
 
 Peer::Peer() :
+	_id(),
+	_ipv4p(),
+	_ipv6p(),
+	_lastUnicastFrame(0),
+	_lastMulticastFrame(0),
+	_vMajor(0),
+	_vMinor(0),
+	_vRevision(0),
 	_dirty(false)
 {
 }
@@ -37,37 +45,44 @@ Peer::Peer() :
 Peer::Peer(const Identity &myIdentity,const Identity &peerIdentity)
 	throw(std::runtime_error) :
 	_id(peerIdentity),
+	_ipv4p(),
+	_ipv6p(),
+	_lastUnicastFrame(0),
+	_lastMulticastFrame(0),
+	_vMajor(0),
+	_vMinor(0),
+	_vRevision(0),
 	_dirty(true)
 {
 	if (!myIdentity.agree(peerIdentity,_keys,sizeof(_keys)))
 		throw std::runtime_error("new peer identity key agreement failed");
 }
 
-void Peer::onReceive(const RuntimeEnvironment *_r,Demarc::Port localPort,const InetAddress &fromAddr,unsigned int latency,unsigned int hops,Packet::Verb verb,uint64_t now)
+void Peer::onReceive(const RuntimeEnvironment *_r,Demarc::Port localPort,const InetAddress &remoteAddr,unsigned int hops,Packet::Verb verb,uint64_t now)
 {
 	if (!hops) { // direct packet
-		WanPath *wp = (fromAddr.isV4() ? &_ipv4p : &_ipv6p);
-
+		WanPath *wp = (remoteAddr.isV4() ? &_ipv4p : &_ipv6p);
 		wp->lastReceive = now;
-		if (verb == Packet::VERB_FRAME)
-			wp->lastUnicastFrame = now;
-		if (latency)
-			wp->latency = latency;
 		wp->localPort = localPort;
 		if (!wp->fixed)
-			wp->addr = fromAddr;
+			wp->addr = remoteAddr;
+		_dirty = true;
+	}
 
+	if (verb == Packet::VERB_FRAME) {
+		_lastUnicastFrame = now;
+		_dirty = true;
+	} else if (verb == Packet::VERB_MULTICAST_FRAME) {
+		_lastMulticastFrame = now;
 		_dirty = true;
 	}
 }
 
-bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,bool relay,Packet::Verb verb,uint64_t now)
+bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,uint64_t now)
 {
 	if ((_ipv6p.isActive(now))||((!(_ipv4p.addr))&&(_ipv6p.addr))) {
 		if (_r->demarc->send(_ipv6p.localPort,_ipv6p.addr,data,len,-1)) {
 			_ipv6p.lastSend = now;
-			if (verb == Packet::VERB_FRAME)
-				_ipv6p.lastUnicastFrame = now;
 			_dirty = true;
 			return true;
 		}
@@ -76,8 +91,6 @@ bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,b
 	if (_ipv4p.addr) {
 		if (_r->demarc->send(_ipv4p.localPort,_ipv4p.addr,data,len,-1)) {
 			_ipv4p.lastSend = now;
-			if (verb == Packet::VERB_FRAME)
-				_ipv4p.lastUnicastFrame = now;
 			_dirty = true;
 			return true;
 		}
@@ -86,6 +99,17 @@ bool Peer::send(const RuntimeEnvironment *_r,const void *data,unsigned int len,b
 	return false;
 }
 
+void Peer::onSent(const RuntimeEnvironment *_r,bool relay,Packet::Verb verb,uint64_t now)
+{
+	if (verb == Packet::VERB_FRAME) {
+		_lastUnicastFrame = now;
+		_dirty = true;
+	} else if (verb == Packet::VERB_MULTICAST_FRAME) {
+		_lastMulticastFrame = now;
+		_dirty = true;
+	}
+}
+
 bool Peer::sendFirewallOpener(const RuntimeEnvironment *_r,uint64_t now)
 {
 	bool sent = false;