Hooking Linux machine to secondary router/ISP: how to setup routing correctly?

Question

Abstract:

I have (I presume routing) issues when adding a secondary ISP to my existing network setup: incoming traffic through Router1 is not answered, but local traffic and incoming through Router0 is working fine.

How can I keep the parts that currently function well working, while making incoming traffic through Router1 work?

Elaboration:

I've sketched a diagram below with the bare essentials of the situation (in practice there are more devices on each LAN, but they do not matter).

This is the situation:

I have two internal networks: LAN0 is 192.168.x.0/24 and LAN1 is 192.168.y.0/24. Both work fine for internal traffic (for instance http using cURL).
LAN0 has always been connected through Router0 and ISP0 to the Internet.
LAN1 always had Router1, but is now connected through ISP1 to the Internet.
Machines only on LAN0 and having a default route through Router0 work fine for outgoing and incoming traffic.
Machines only on LAN1 and having a default route through Router1 work fine for outgoing and incoming traffic.
Internal traffic on LAN0 and LAN1 has always worked fine.
Incoming traffic through Router1 for WindowsB arrives correctly: I can connect to it over RDP from WindowsC.
Incoming traffic through Router1 for LinuxB arrives (according to tcpdump), but not answered back as a curl http://e.f.g.h fron LinuxC shows with a tcpdump on LinuxB shows:

It shows only packets that - according to the tcpdump output format - have a SYN flag set:

LinuxB:/tmp/LinuxB.eth1.80 # tcpdump -i eth1 'port 80'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
13:35:19.489779 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287047182 ecr 0,sackOK,eol], length 0
13:35:19.788841 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287047478 ecr 0,sackOK,eol], length 0
13:35:19.888835 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287047578 ecr 0,sackOK,eol], length 0
13:35:19.989412 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287047678 ecr 0,sackOK,eol], length 0
13:35:20.089685 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287047778 ecr 0,sackOK,eol], length 0
13:35:20.190836 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287047877 ecr 0,sackOK,eol], length 0
13:35:20.392123 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1287048072 ecr 0,sackOK,eol], length 0
13:35:20.693692 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,sackOK,eol], length 0
13:35:21.197162 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,sackOK,eol], length 0
13:35:22.204134 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,sackOK,eol], length 0
13:35:24.115961 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,sackOK,eol], length 0
13:35:27.852374 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,sackOK,eol], length 0
13:35:31.967049 IP i.j.k.l.57512 > 192.168.y.2.http: Flags [S], seq 816356596, win 65535, options [mss 1460,sackOK,eol], length 0

This is the LinuxB route table:

LinuxB:/tmp/LinuxB.eth1.80 # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.x.1     0.0.0.0         UG    0      0        0 eth0
loopback        *               255.0.0.0       U     0      0        0 lo
link-local      *               255.255.0.0     U     0      0        0 eth0
192.168.x.0     *               255.255.255.0   U     0      0        0 eth0
192.168.x.0     *               255.255.255.0   U     0      0        0 eth1

Since connecting over RDP from WindowsC to WindowsB works fine, I resume this is indeed a routing problem. This is the WindowsB route table:

C:\temp>route print
===========================================================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x2 ...00 0c 29 35 77 e1 ...... AMD PCNET Family PCI Ethernet Adapter - Packet Scheduler Miniport
0x3 ...00 0c 29 35 77 eb ...... VMware Accelerated AMD PCNet Adapter - Packet Scheduler Miniport
===========================================================================
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0      192.168.x.1     192.168.x.4      10
          0.0.0.0          0.0.0.0      192.168.y.1     192.168.y.4       5
        127.0.0.0        255.0.0.0        127.0.0.1       127.0.0.1       1
      192.168.x.0    255.255.255.0      192.168.x.4     192.168.x.4      10
      192.168.x.4  255.255.255.255        127.0.0.1       127.0.0.1      10
    192.168.x.255  255.255.255.255      192.168.x.4     192.168.x.4      10
      192.168.y.0    255.255.255.0      192.168.y.4     192.168.y.4      10
      192.168.y.4  255.255.255.255        127.0.0.1       127.0.0.1      10
    192.168.y.255  255.255.255.255      192.168.y.4     192.168.y.4      10
        224.0.0.0        240.0.0.0      192.168.x.4     192.168.x.4      10
        224.0.0.0        240.0.0.0      192.168.y.4     192.168.y.4      10
  255.255.255.255  255.255.255.255      192.168.x.4     192.168.x.4       1
  255.255.255.255  255.255.255.255      192.168.y.4     192.168.y.4       1
Default Gateway:       192.168.y.1
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
          0.0.0.0          0.0.0.0      192.168.y.1       5
          0.0.0.0          0.0.0.0      192.168.x.1      10

So how can I get the routing on LinuxB to be like this:

keep default route on LinuxB to 192.168.x.1 so outgoing traffic keeps using Router0/ISP0
keep answering incoming requests coming from LAN0 on LAN0
keep answering incoming requests coming from LAN1 on LAN1
keep answering incoming requests through Router0 (a.b.c.d/192.168.x.1) via 192.168.x.1
start answering incoming requests through Router1 (e.f.g.h/192.168.y.1) via 192.168.y.1
bonus: have Router1 fail-over or load-balance with Router0

Postscript:

The PNG image below is generated on UML text through the free online PlantUML engine. If you want to see the original UML text, paste the PNG image link into this PlantUML form, then press Submit.

enter image description here

marc.fargas · Answer 1 · 2014-07-25T21:49:17.973

I had a shell script for doing something like that a long-long time ago but, sorry, could find it. So I can only give you the pointers to the solutions I implemented back then. I'm writting mostly from memory so some examples are missing:

I had one routing table per uplink (ip route ... table 101, ip route ... table 102). That goes into /etc/iproute2/rt_tables.

101 isp1 102 isp2

You need to setup those tables also:

ip route add default via $Gateway1 dev $Interface1 table isp1 ip route add default via $Gateway2 dev $Interface2 table isp2

#Don't forget the default table:

ip route add default via $DefaultGateway dev $DefaultInterface
Enable iptables connection tracking (modprobe nf_conntrack)
Set an iptables rule for NEW incoming connections to -j MARK the packets somehow (ie: 0x201, 0x202)
Set an ip rule that ensure that traffic going out trhough an interface uses the right routing table

ip rule add from $Ip1 table isp1 ip rule add from $Ip2 table isp2
Set an ip rule (ip rule add ...) stating that "packets marked 0x201 shall lookup routing table 201", one rule for every uplink.

With everything in place you should be able to receive and initiate connections with any wan uplink and even balance outgoing connections.

Those would be the basics. Iptables + "ip route" + "ip rule" and you're good to go.

Hooking Linux machine to secondary router/ISP: how to setup routing correctly?

1 Answers1