12

Short Version

My home network is pure gigabit with devices that all support jumbo frames up to at least ~9000 bytes. Increasing the MTU jumbo frame setting on the Synology to 6000 (bytes) increases performance (810Mbps write and 945Mbps read). Setting the value to 7000 destroys only read performance (which decreases all the way down to 4Mbps); write performance stays speedy.

This is unexpected because most jumbo frame problems don't have a directionality associated with them and are typically all or nothing (packets are dropped at a switch no matter where they came from). There doesn't appear to be any IP fragmentation going on at all, but the TCP layer is really unhappy. What could cause this asymmetric/flaky behavior and how can I fix it to support the full 9000 byte MTU all of my equipment is supposed to support?

Long Version

These are my edited notes taken while trying to figure this out.

Client

Realtek PCIe GBE Family Controller RTL8167
Jumbo Frame: 9KB MTU

$ netsh interface ipv4 show subinterfaces
   MTU  MediaSenseState   Bytes In  Bytes Out  Interface
------  ---------------  ---------  ---------  -------------
  9198                1   32501506   11275394  Local Area Connection

(appears 9198 doesn't include the 14 byte ethernet header)

$ ping -l 1500 -f 192.168.1.84

(observed with Wireshark running on Client; all sizes are wire byte sizes)
[9213, ∞] not sent by host (would require fragmenting)
[9019, 9212] sent but no response
[9015, 9018] fragmented IP response
[42, 9014] unfragmented IP
[0, 41] ? (unable to generate since eth+IP+ICMP headers = 14+20+8 = 42 bytes)

Router (switch portion)

Asus RT-AC68U -- Firmware 3.0.0.4.378_4585
Enable Jumbo Frame: "Enable"
Can't figure out what jumbo frame size it actually supports, appears to be at least 9000

It fragments ping requests from Client right at 1514 bytes (but pinging the router might be triggering its WAN router behavior instead of its LAN switch behavior?)

Unmanaged Switch

TP-LINK TL-SG1008D
Jumbo Frames (spec sheets): 9KB (their website says 15KB but it looks like a different device)

Server

Synology DS1815+ -- DSM 5.2-5565 Update 1
Jumbo Frame: 9000

File-read packets from Synology to Client
Size: most are 9014 bytes (in both directions)
IP Flags: Don't fragment
Wireshark discovered: TCP Spurious Retransmission, TCP Previous segment not captured, TCP Out-Of-Order, TCP Fast Retransmission, and normal (9014 byte) packets
SMB2-over-NetBIOS-protocol packets read response read length: 65,536 (~8 TCP segments) 

$ ifconfig
bond0     Link encap:Ethernet  HWaddr --:FF
          inet addr:192.168.1.84  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addrs: --/64 Scope:Link, --/64 Scope:Global, --/64 Scope:Global
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:9000  Metric:1
          RX packets:lots errors:85 dropped:0 overruns:0 frame:85
          TX packets:lots errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:237 GiB  TX bytes:117 GiB

eth2      Link encap:Ethernet  HWaddr --:00
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:9000  Metric:1
          RX packets:lots errors:19 dropped:0 overruns:0 frame:19
          TX packets:lots errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:236 GiB  TX bytes:83 GiB

eth3      Link encap:Ethernet  HWaddr --FF
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:9000  Metric:1
          RX packets:lots errors:66 dropped:0 overruns:0 frame:66
          TX packets:lots errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1 GiB  TX bytes:33 GiB

eth2 and eth3 are bonded using Adaptive Load Balancing (no switch support)

$ ping -c 5 -s 1500 192.168.1.82

(observed with Wireshark running on Client; all sizes are wire byte sizes)
[9019, ∞] request sent, response sent, response not received
[9015, 9018] fragmented IP request (probably fragmented by Synology, busybox ping doesn't have a no-fragment option so it's hard to tell)
[60, 9014] unfragmented IP
[0, 59] ? (unable to generate because busybox ping puts at a minimum 18 bytes plus the 42 byte headers)

Misc Data

  • Changing the Client MTU down to 8KB didn't help
  • The Server's read speed falls off a cliff when changing the Server's MTU from 6000 (great, 945Mbps) to 7000 (terrible, 4Mbps)
  • The Server's write speed is basically unaffected at all Server MTU settings (always between 700 and 825 Mbps)
  • The Synology has a bonded network (2 of the 4 ports)
  • Cables are all Cat6 or Cat5e
gparyani
  • 1,881
colithium
  • 1,075

1 Answers1

2

Update the Firmware

In my experience, Synology fixes a lot of issues in each firmware release and the one you're running is nearly four years old. I have not read the release notes, but there seems to be lots of opportunity for a Jumbo frame bug to have been fixed since then.

Test with a Direct Connection

Connect your test machine directly to the Synology (assign static IPs on the same subnet) with new patch cables and re-run your tests. This will eliminate the cabling and switches as well as any other equipment and configuration issues. If the problem remains, run your tests with another computer. If it still remains it's certainly the NAS.

If the problem goes away during the direct-connect test, then try replacing the switch first, then the cabling. You haven't shown the connections so I'm assuming just the TPLINK between the test machine and the NAS.

Community
  • 1
Jens Ehrich
  • 923