I have seen in many places this iptables rule iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu to deal with Path MTU Discovery issues.
From my understanding, PMTU may differ in multiple paths (say A->B has PMTU 1400, A->C has PMTU 1350). So how does iptables/kernel calculates PMTU?
It asks the routing table.
Typically the rule is meant to be used on the same router that is at the edge of the path with the lower MTU.
For example, if you have router with a PPPoE interface or a site-to-site GRE tunnel (with its lower MTU), then you'd have this rule on that particular router, not on the originating host, and it would tweak the advertised MSS according to what interface it's about to send the SYN packet through.
Adjusting the MSS for narrower links further down the road is not really within the scope of this module – indeed it's meant for situations where standard PMTUD is not working for the problematic paths and therefore distant hosts can't find out the correct MTU. (Usually it's meant to work around PMTUD issues that occur on the other end of the link, i.e. remote hosts trying to send too big packets towards you.)
(In theory the module could also query the kernel's "route cache" which remembers the discovered path MTU for each address it has recently communicated with, but of course this would rely on PMTUD working correctly and "Too big" errors arriving, and the whole point of this netfilter module is to deal with situations where PMTUD is not working at all.)
It doesn't calculates the PMTU, it discovers it.
PMTUD is a protocol that requires a sender to discover the largest MTU by sending repeatedly smaller packets until it stops receiving an ICMP "Destination Unreachable" response. This is inefficient, but avoids fermentation.
TCP MSS is a more recent protocol used with TCP at layer 4.
While establishing a new TCP connection, a three-way handshake is performed. Each device inserts its MSS into TCP headers, so in this sense, it’s announcing its MSS to the remote device, without any negotiation of mutually acceptable values.
The problem that needs solving is that a device will only know the MTU, and therefore the MSS, of its local link, but will not know about a lower MSS on a link somewhere along the path.
The solution is to configure the routers to rewrite the MSS.
A Cisco router, for example, will have this command configured on a tunnel:
ip tcp adjust mss 1436
When TCP traffic, such as the three-way handshake, passes across the tunnel, the router will see the MSS is set to 1460 in the TCP header and will rewrite this to 1436, and the host at the other end will adjust the maximum payload for this connection accordingly.
This option was previously required to be enabled through direct iptables rules. However, firewalld for example now enables it as an option that applies automatically to all rules.
This applies only to TCP, not UDP, and requires routers to participate in the message routing.
Reference : TCP MSS Clamping in Firewalld.
