I'm in the process of upgrading my Ethernet to 10 Gbps to speed up the connectivity of my local network. Regretfully, a machine on my network has ran out of PCIe slots, so a native 10 GbE PCIe adapter is not an option. A USB adapter is the only feasible alternative - as 5 Gbps Ethernet switches and USB adapters are hard to find and expensive, the decision was made to run the server with two 2.5 Gbps Ethernet to USB 3 adapters based on the Realtek RTL8156 chipset, which is readily-available. Then, they're together at the Ethernet level using link aggregation in a LAG/LACP, and connected to a 2.5/10 Gbps switch.
These USB adapters can be recognized by Linux as the following. I've also determined beforehand that both adapters can operate independently at 2.5 Gbps by assigning an IP address to each NIC and running a iperf3 test.
$ dmesg
[ 5.118103] usb 4-4: new SuperSpeed USB device number 2 using xhci_hcd
[ 5.138434] usb 4-4: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.04
[ 5.138436] usb 4-4: New USB device strings: Mfr=1, Product=2, SerialNumber=6
[ 5.138438] usb 4-4: Product: USB 10/100/1G/2.5G LAN
[ 5.138439] usb 4-4: Manufacturer: Realtek
[ 5.138440] usb 4-4: SerialNumber: 401300ÿÿÿÿ
[ 6.970319] cdc_ncm 4-4:2.0: MAC-Address: 00:e0:4c:68:10:e6
[ 6.970325] cdc_ncm 4-4:2.0: setting rx_max = 16384
[ 6.970356] cdc_ncm 4-4:2.0: setting tx_max = 16384
[ 6.984099] cdc_ncm 4-4:2.0 eth0: register 'cdc_ncm' at usb-0000:0a:00.3-4, CDC NCM (NO ZLP), 00:e0:4c:68:10:e6
lsusb
Bus 004 Device 002: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN
Bus 002 Device 002: ID 0bda:8156 Realtek Semiconductor Corp. USB 10/100/1G/2.5G LAN
The machine in question is running a Linux hypervisor (Proxmox), and two network interfaces are recognized as enx00e04c680152 and enx00e04c6810e6 (I believe udev/systemd generated the names based on their MAC addresses). To bond two interfaces together, on Proxmox I used the following settings:
Linux Bond
* Name: bond0
* Autostart: Yes
* Slaves: enx00e04c680152 enx00e04c6810e6
* Mode: LACP (802.3ad)
* Hash policy: layer3+4
On the switch, I created a Port Channel Group with LACP enabled in Active mode, and selected two 2.5 Gbps ports that were connected to the server. Linux also reports that both card are detected and bonded from its own perspective:
# ip link | grep enx
3: enx00e04c6810e6: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT group default qlen 1000
4: enx00e04c680152: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT group default qlen 1000
Unfortunately, for some reason, Linux cannot establish an LACP link with the switch no matter what, Linux kernel keeps telling me that "No 802.3ad response from the link partner for any adapters in the bond".
# dmesg
[ 1004.491253] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
[ 1034.527234] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
[ 1064.547217] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
The status file /proc/net/bonding/bond0 shows that IEEE 802.3ad is active, but a link is not successfully
established. This can be seen from the fact that the "Aggregator ID" for both NICs are different, and that
both of them entered "churned" state, which essentially means the link failed,
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.1.15-1-pve
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 8a:f5:1c:f4:8b:70
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 1
Actor Key: 10
Partner Key: 1
Partner Mac Address: 00:00:00:00:00:00
Slave Interface: enx00e04c680152
MII Status: up
Speed: 2500 Mbps
Duplex: half
Link Failure Count: 0
Permanent HW addr: 00:e0:4c:68:01:52
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 8a:f5:1c:f4:8b:70
port key: 10
port priority: 255
port number: 1
port state: 77
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1
Slave Interface: enx00e04c6810e6
MII Status: up
Speed: 2500 Mbps
Duplex: half
Link Failure Count: 0
Permanent HW addr: 00:e0:4c:68:10:e6
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 8a:f5:1c:f4:8b:70
port key: 10
port priority: 255
port number: 2
port state: 69
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1
