TCP probe support is implemented, unnecessarily, across the stack and and tcp
packages. It can live entirely in tcp. Additionally, it is only ever set at
initialization time, so support for dynamically adding/removing the probe isn't
necesary.
The probe is getting in the way of adding debugging for b/339664055.
PiperOrigin-RevId: 699330364
Implement the core part. All packets are broadcast-ed to all ports. The next
step will be to implement forwarding and multicast group databases.
PiperOrigin-RevId: 644471256
Having read more GRO kernel code and NAPI, I believe the previous design was
overly complex and resulted in poor performance.
- Netstack GRO uses a `time.Timer` to periodically clear GRO'd packets. It's
got... several problems.
1. The timer can be configured via CLI flag with arbitrary granularity, but
(IIUC) `time.Timer` relies on the netpoller, which [has millisecond
granularity].
2. The timer creates new goroutines when it fires.
3. There's a complex atomic value song and dance to setting up the timer,
canceling it when no packets are pending, and resuming it for incoming
packets.
- Linux GRO doesn't quite work this way.
- Typically, [Linux flushes GRO whenever it can] (unless running with high
HZ; Go preempts goroutines every 10ms so this does not apply). IIUC,
receiving packets triggers the scheduling of a softirq that batch reads as
many packets from a device as possible. softirqs are run as kernel threads,
so this is getting scheduled with jiffy-ish granularity.
- There's no special scheduling of a timer to flush GRO.
In Netstack our "interrupt" is that we return from a poll, then we read
multiple packets at once via recvmmsg/readv/XDP. So we've already got a delay
analogous to "trigger a ksoftirq and wait for the thread to get scheduled."
Thus, we should use zero-timeout GRO that does the following:
- When recvmmsg/poll/etc returns a single packet, just pass it directly and
immediately up the stack.
- When multiple packets are returned, coalesce them with GRO and flush them
without waiting. We never have 1000+ HZ situation.
- We can remove all atomics and locking from GRO, as there will be one
`groDispatcher` per dispatcher goroutine and no timer-spawned goroutines to
synchronize with.
**Performance**: The previous GRO implementation yielded a few percentage points
increase in performance. This is markedly better.
The following is from tcp_benchmark. It is running with host GRO/GSO disabled,
as there's nothing to GRO when the host does it for us. The RecvMMsg dispatcher
is used, as the PacketMMap dispatcher does not return multiple packets at once
(RecvMMsg averages 8 per syscall in these benchmarks).
```
│ /tmp/old.log │ /tmp/new.log │
│ Mb/s │ Mb/s vs base │
TCP/role=server/host-gso=false/host-gro=false 1.764k ± 2% 2.139k ± 2% +21.29% (p=0.000 n=20)
```
PiperOrigin-RevId: 622366744
This is effectively a rollback of cl/450976957. The original motivation never
panned out, and it's easier to work with the lists. They also are easier to
avoid allocations with.
PiperOrigin-RevId: 531020857
The benefits of PacketBufferPtr never materialized and it makes the type
difficult to work with, e.g. it can't be used with go_generics to make a
(performant) list.
This is effectively a rollback of cl/480518221.
PiperOrigin-RevId: 530954630
A 20,000 ns interval yields (depending on the run) a roughly 10-30% throughput
boost in the TCP benchmark. Under heavier load with more contention the boost
should be larger.
As noted in the TODOs, there are a large number of low-hanging optimizations to
be made. This CL just implements a simple form of GRO.
Tested via existing tests (there should be no user-visible behavior changes).
Unit tests coming in a child CL.
GRO is off by default and so users are unaffected unless they enable it
explicitly.
PiperOrigin-RevId: 489313603
Checksum capabilities are logically separate from tcpip header definitions
and operations. It makes sense to extract this logic into its own package.
This is also necessary to avoid circular dependencies with bufferv2.
PiperOrigin-RevId: 473096480
This change has significant performance implications. bufferv2 is reference
counted and pooled, which alleviates heap/GC pressure. Below are the results
from running the iperf benchmark.
HEAD:
BenchmarkIperf/operation.Upload-16 1552 ns/op 46.6GiB total allocations
BenchmarkIperf/operation.Download-16 1114 ns/op 68.6GiB total allocations
w/ change:
BenchmarkIperf/operation.Upload-16 1139 ns/op (-27%) 1.41GiB total allocations (-97%)
BenchmarkIperf/operation.Download-16 753.2 ns/op (-33%) 706MiB total allocations (-99%)
PiperOrigin-RevId: 462453185
This change implements AddMulticastRoute and the requisite routing logic.
Subsequent changes will still be needed to:
1. Emit events for missing route or unexpected input interface
2. Implement DelRoute
3. Implement GetRouteStats
Updates #7338.
PiperOrigin-RevId: 451026594
Kevin KrakauerandgVisor bot
Nayana BidariandgVisor bot
Andrei VaginandgVisor bot
Fabricio VoznikaandgVisor bot
Jing ChenandgVisor bot
Adin ScannellandgVisor bot
Andrei Vagin
Ayush RanjanandgVisor bot
Lucas ManningandgVisor bot
Nate HurleyandgVisor bot