3

Hardware setup

  • NAS: Core i3 7100, Intel X540T2 10GbE
  • Switch: Mikrotik CRS305
  • PC: Ryzen 7 2700, Intel X540T2 10GbE
  • Cables: 2x Cat 7 cables

Software Setup

  • NAS: Ubuntu 20.04 
    alexwang@alex-nas2 ~ (master)> iperf3 --version
iperf 3.7 (cJSON 1.5.2)
Linux alex-nas2 5.4.0-47-generic #51-Ubuntu SMP Fri Sep 4 19:50:52 UTC 2020 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, SCTP, TCP congestion algorithm 
setting, sendfile / zerocopy, socket pacing, authentication

    alexwang@alex-nas2 ~ (master) [255]> sudo sysctl -p
net.ipv4.ip_forward = 1
kernel.perf_event_paranoid = -1
vm.nr_hugepages = 128
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
net.ipv4.tcp_congestion_control = htcp
net.ipv4.tcp_mtu_probing = 1
net.core.default_qdisc = fq
  • PC: Dual boot Windows 10 2004 and Arch Linux latest version(20200911)
    • Windows 10
      • WSL2 Ubuntu 20.04
      • MTU 1500, TCPWindowSize=1M, 
        alexwang@Alex-PC /m/c/U/alexwang> iperf3 --version
iperf 3.7 (cJSON 1.5.2)
Linux Alex-PC 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, SCTP, TCP congestion algorithm 
setting, sendfile / zerocopy, socket pacing, authentication
      • test with iperf3 -c NAS_IP
    • Arch Linux - iperf 3.7 - test with iperf3 -c NAS_IP - MTU 1500, - TCP tuning parameter are all default values of Kernel 5.4.64
  • PC to NAS RTT:
alexwang@Alex-PC /m/c/U/alexwang> ping 192.168.201.50
PING 192.168.201.50 (192.168.201.50) 56(84) bytes of data.
64 bytes from 192.168.201.50: icmp_seq=1 ttl=63 time=0.598 ms
64 bytes from 192.168.201.50: icmp_seq=2 ttl=63 time=0.550 ms
64 bytes from 192.168.201.50: icmp_seq=3 ttl=63 time=0.528 ms
64 bytes from 192.168.201.50: icmp_seq=4 ttl=63 time=0.612 ms

Expected Behavior

iperf3 test speed from PC to NAS should be the same (nearly 10Gbps) regardless of what operating system I am running on my PC(Windows 10 or Arch Linux), since the hardwares are the same.

Actual Behavior

When I boot into Arch Linux on my PC, I can get 10Gbps speed(event with -P 1). But in Windows 10 I can only get about 2.5Gbps with -P 1, and at maximum speed of 6Gbps with -P 8. I hope Windows can achieve 10Gbps with single connection, too.

Already tried

  • Do not use WSL, use iperf3 windows prebuilt binary from https://iperf.fr/iperf-download.php => Same speed
  • Increase MTU to 5000(on NAS, PC and switch ports) => Sorry I forgot the actual numbers but not getting faster.(Should not be MTU problem since Arch Linux with 1500MTU can still run at 10Gbps)
  • Set Windows TCPWindowSize to 1M => We can get 6Gbps at 8 connections(previously 4Gbps at 8 connections), but increasing it to 2M makes no difference
  • Increase NAS TCP receive window size => (Clearly it makes no difference because the bottleneck is at my PC not NAS)
  • Change cables to a new one. => no difference
  • Swap the NIC between my PC and NAS. => no difference
  • Disable Windows TCP autotuning => Slower speed(1.8Gbps with single connection)
  • Manually set TCP windows size using iperf3(iperf3 -w 256k) => Slower speed(1Gbps with single connection. Of course it would be slower because iperf3 with choose a much bigger window size). But iperf3 does not support setting window size to high than about 410k, so I can't try high window size. 
    alexwang@Alex-PC /m/c/U/alexwang [1]> iperf3 -c 192.168.201.50 -w 420k
Connecting to host 192.168.201.50, port 5201
iperf3: error - socket buffer size not set correctly

Other observations that may be related

  • When running iperf3 -c NAS_IP -R on my PC running windows, I can see the whole system gets slower, the System process in task manager takes up 10% CPU.
  • Local test can achieve 15Gbps 
    # on Windows PC powershell
.\iperf3 -s -p 41111
# also on Windows PC powershell, 192.168.201.60 is my Windows PC Intel X540 NIC IP
.\iperf3 -c 192.168.201.60 -p 41111

My guess

  • I'm not familiar with Windows so I guess I missed some of the Windows TCP tunnings that cause the slow speed but I don't know which one.
  • I'd like to know how to solve the problem, or at least other things I should try or learn to get closer to the problem.

Updates

  • update1. fix format
Alex Wang
  • 131

0 Answers0