Beyond Gigabit (On 1Gb Ethernet) - VLAN Bonding

Congratulation, you've got a personal gigabit internet pipe! Just plug your server into the ISP's modem and you'll get 1Gbps, right?

Nope.

You're realistically looking at ~925-950Mbps at the network layer, as 1GbE is only at the data link layer.

There's two solutions: upgrade to 10Gb hardware, or use bonding. The former is going to be a hard sell to your ISP when you're only paying for 1Gbps to begin with, so we're left with bonding as the remaining choice. (And if you're using 10GbE, you probably still want bonding for redundancy.)

Network Layout

Switch

In my case I'm using VID 2372 and 2662 for the two WAN ports.

Don't put the two WAN ports on the same VLAN ID, unless you feel like testing your modem's loopback detection.

/etc/network/interfaces

auto lo
iface lo inet loopback

auto enp1s0
iface enp1s0 inet manual
bond-master bond0

auto enp2s0
iface enp2s0 inet manual
bond-master bond0

auto enp3s0
iface enp3s0 inet manual
bond-master bond0

auto bond0
iface bond0 inet manual
bond-mode balance-rr
# bond-mode 802.3ad
bond-slaves enp1s0 enp2s0 enp3s0
bond-miimon 100
bond-downdelay 200
bond-updelay 200
# bond-lacp-rate 1
bond-xmit-hash-policy layer2
# bond-xmit-hash-policy layer3+4
hwaddress ether 00:20:91:80:23:AD

# WAN-Port-01
auto vlan2372
iface vlan2372 inet manual
bond-master vbond0
vlan-raw-device bond0
hw-mac-address 00:20:91:23:72:00

# WAN-Port-02
auto vlan2662
iface vlan2662 inet manual
bond-master vbond0
vlan-raw-device bond0
hw-mac-address 00:20:91:26:62:00

auto vbond0
iface vbond0 inet dhcp
bond-mode balance-rr
bond-slaves vlan2372 vlan2662
# bond-xmit-hash-policy layer3+4
hwaddress ether 00:20:91:15:9B:CD

Hashing

Layer 3+4 hashing is not needed as we're connecting to two physically different ports on the modem. It probably wouldn't hurt to use though.

Static MAC Addresses

I would recommend assigning a MAC address, otherwise it will be confusing to tell if the bonding is working, or if the system is just using the first NIC by itself (e.g. in PXE). Also, if the order of network cards change, the bonded interface will default to a different MAC address.

LACP

If possible, try using 802.3ad (LACP) instead of round-rr. (My switch does not load balance properly, and ends up clustering all the incoming traffic to a single port in a LACP group.) If a PEBKAC cabling incident happens, you'll only end up spamming LACPDU frames to the wrong port.

/etc/sysctl.conf

Apparently there's a "known bug" in the Linux kernel with multiple bonded links and IPv6, according to a random person on IRC. (This report is the best reference I could find.)

net.ipv6.conf.vbond0.accept_dad=0
net.ipv6.conf.vlan2372.use_tempaddr=0
net.ipv6.conf.vlan2372.autoconf=0
net.ipv6.conf.vlan2662.use_tempaddr=0
net.ipv6.conf.vlan2662.autoconf=0

I'm also turning off SLAAC on the VLANs, since I don't want to accidentally use them instead of the bonded interface.

Results

dave@intel:~$ iperf3 -c iperf.he.net --zerocopy --parallel 8 --reverse --omit 30 -t 60 --format m --interval 60
Connecting to host iperf.he.net, port 5201
Reverse mode, remote host iperf.he.net is sending
[  4] local 2a07:1c44:14b4:: port 58112 connected to 2001:470:0:238::2 port 5201
[  6] local 2a07:1c44:14b4:: port 58114 connected to 2001:470:0:238::2 port 5201
[  8] local 2a07:1c44:14b4:: port 58116 connected to 2001:470:0:238::2 port 5201
[ 10] local 2a07:1c44:14b4:: port 58118 connected to 2001:470:0:238::2 port 5201
[ 12] local 2a07:1c44:14b4:: port 58120 connected to 2001:470:0:238::2 port 5201
[ 14] local 2a07:1c44:14b4:: port 58122 connected to 2001:470:0:238::2 port 5201
[ 16] local 2a07:1c44:14b4:: port 58124 connected to 2001:470:0:238::2 port 5201
[ 18] local 2a07:1c44:14b4:: port 58126 connected to 2001:470:0:238::2 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-60.00  sec  1.02 GBytes   146 Mbits/sec
[  6]   0.00-60.00  sec  1.02 GBytes   146 Mbits/sec
[  8]   0.00-60.00  sec  1017 MBytes   142 Mbits/sec
[ 10]   0.00-60.00  sec  1.00 GBytes   143 Mbits/sec
[ 12]   0.00-60.00  sec  1.03 GBytes   148 Mbits/sec
[ 14]   0.00-60.00  sec  1.03 GBytes   148 Mbits/sec
[ 16]   0.00-60.00  sec  1.04 GBytes   149 Mbits/sec
[ 18]   0.00-60.00  sec  1.05 GBytes   151 Mbits/sec
[SUM]   0.00-60.00  sec  8.20 GBytes  1174 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  1.02 GBytes   147 Mbits/sec  703             sender
[  4]   0.00-60.00  sec  1.02 GBytes   147 Mbits/sec                  receiver
[  6]   0.00-60.00  sec  1.02 GBytes   147 Mbits/sec  823             sender
[  6]   0.00-60.00  sec  1.02 GBytes   147 Mbits/sec                  receiver
[  8]   0.00-60.00  sec  1019 MBytes   142 Mbits/sec  609             sender
[  8]   0.00-60.00  sec  1020 MBytes   143 Mbits/sec                  receiver
[ 10]   0.00-60.00  sec  1.00 GBytes   143 Mbits/sec  1455             sender
[ 10]   0.00-60.00  sec  1.00 GBytes   144 Mbits/sec                  receiver
[ 12]   0.00-60.00  sec  1.03 GBytes   148 Mbits/sec  798             sender
[ 12]   0.00-60.00  sec  1.03 GBytes   148 Mbits/sec                  receiver
[ 14]   0.00-60.00  sec  1.03 GBytes   148 Mbits/sec  1233             sender
[ 14]   0.00-60.00  sec  1.03 GBytes   148 Mbits/sec                  receiver
[ 16]   0.00-60.00  sec  1.04 GBytes   150 Mbits/sec  575             sender
[ 16]   0.00-60.00  sec  1.04 GBytes   150 Mbits/sec                  receiver
[ 18]   0.00-60.00  sec  1.06 GBytes   151 Mbits/sec  577             sender
[ 18]   0.00-60.00  sec  1.06 GBytes   151 Mbits/sec                  receiver
[SUM]   0.00-60.00  sec  8.21 GBytes  1175 Mbits/sec  6773             sender
[SUM]   0.00-60.00  sec  8.22 GBytes  1177 Mbits/sec                  receiver

iperf Done.

tl;dr: 1177Mbps through a 1GbE modem.