0

Dell PowerEdge T110 II server (Broadcom BCM5722 using tg3 driver, Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz, Almalinux 8.5) has asymmetric iperf3 performance. When it is the server iperf3 tests at 940 Mbps, but when it is the client iperf3 tests at 740 Mbps (AlmaLinux, X11 running) or 800 Mbps (SystemRescue, no X11, or AlmaLinux without X11 running).

Variables which have been eliminated:

selinux  (same speed on/off)
iptables (same speed on/off)
gso/tso  (slower when they are off, default was on)
autonegotiation (
tx-checksum-ipv4 is on
patch cables
wifi/router (same speed using a dumb 1G switch)
CPU speed (same speed with "performance" as "conservative" governors)
user vs. root (same speed, either way)
ethtool Ring parameters (were originally RX 200, TX 511, but making both
   511 did not change the speed)
/proc/sys/net/core/netdev_max_backlog (was 1000, tried 8333: no change 
  in speed)

iDrac is configured but ipmi over the network is disabled.

ifconfig shows no errors, drops, etc.

eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.121  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::a598:26ae:335a:d435  prefixlen 64  scopeid 0x20<link>
        ether d0:67:e5:f3:73:8c  txqueuelen 1000  (Ethernet)
        RX packets 2496273  bytes 2558706885 (2.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6131153  bytes 8785383703 (8.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16

The only hint of a solution I see is that turning off X11 made it run slightly faster. So perhaps some system parameter tweak related to interrupts would help?

mathog
  • 191
  • 1
  • 1
  • 4

1 Answers1

0

I have 2 Dell PowerEdge T110 II servers with the BCM5722 NICs and have exactly the same issue. RX speed is nearly full gigabit (940Mbps according to iperf3), but TX speed is 787Mbps. There are no dropped packets or framing errors to account for the loss of speed. These servers also have a RTL8168 NIC. With the realtek NIC, and using the same patch cables, I get 940Mbps up and down.

One server runs Debian 11 and the other runs ESXi (vSphere 6.7). Both Debian and VMware use the tg3 driver. Ethtool reports both systems have the same BCM firmware (firmware-version: 6.2.1 bc 5722-v3.11). I could not find more recent firmware on Dell's site.

I have been unable to improve the TX performance via ethtool or sysctl tweaks.

According to this Dell forum moderator, the BCM5722 does not support TOE (https://www.dell.com/community/PowerEdge-Hardware-General/Network-cards-on-Poweredge-T110-ii/m-p/4398275). That is confirmed by the technical specs here: https://www1.la.dell.com/content/products/productdetails.aspx/nic_broadcom_5722.

Lack of a TCP offload would explain why the TX is below the RX speed. For me, the NIC has been rock stable and the speed isn't that bad for such an old chipset. I don't think it's possible to fix this without a firmware update. A workaround is simply to use a different NIC that supports TOE (like the Realtek mentioned above).

Jeff
  • 1