Background
I have two lab machines, which I'll refer to as A and B, running RHEL 8.0.
I SSH into A and B from my Windows desktop over a 1 GbE link, whose network interface I'll refer to as eth0.
A and B also have a direct 10 GbE fiber link between them (directly connected, no switches or routers involved). I'll refer to this link's network interface as eth1. The IP addresses of these 10 GbE network interfaces on A and B are 192.168.0.1/24 and 192.168.0.2/24, respectively.
I am using iperf3 v3.5 to test unidirectional (B --> A) UDP throughput. The MTU is set to 1500, and each frame's payload is 1500 bytes so that the frames are as large as they can be without fragmentation.
With firewalld stopped, I get throughput of about 7.00 Gbps and no frame drops.
If I start firewalld, put eth1 into zone public, and add the appropriate UDP port (5201) to zone public, I get throughput of about 6.43 Gbps and no frame drops.
However, if I put eth1 into zone trusted (which passes all traffic without question), I start seeing massive frame drops (about 15%).
On both A and B, CPU 18 is isolated so that it is available for exclusive use by iperf3.
iperf3 server invocation (on A) is:
taskset -c 18 iperf3 -s -V --udp-counters-64bit
iperf3 client invocation (on B) is:
taskset -c 18 iperf3 -u -V -b 0 --udp-counters-64bit -t 120 -c 191.168.0.1 -l 1472
My Question
Why do I see a large number of frames dropped when using firewalld zone trusted (as opposed to public, which works fine)?
Additional Notes Based on Comments Received
COMMENT
Is iperf3 or firewalld hitting 100% cpu usage on that core? Have you tried using dpdk-iperf? Do you get this behavior in reverse too, from A->B? – Cpt.Whale
RESPONSE
When using zone public, iperf3 runs at about 88% CPU utilization on the receiving side and at about 95% CPU utilization on the sending side.
When using zone trusted, iperf3 runs at about 94% CPU utilization on the receiving side and at about 95% CPU utilization on the sending side. However... When using zone trusted, there is another high-CPU-utilization process on the receiving side named ksoftirqd. It runs on a different core and saturates that core at 100% CPU utilization.
firewalld does not consume enough CPU to appear in the list.
I have not tried dpdk-iperf.
Yes, the same behavior occurs in the reverse direction. (The hardware and configuration are identical, so this is expected.)
