When configuring a Linux host with multiple interface, each with its own default gateway, ensuring route symmetric is a bet challenging for any given pair of endpoint. Symmetric routing means that traffic leaves the hosts from a path (egress traffic) and come back from the same path (ingress traffic).

Of course, routing on a multihomed host is straightforward when each interface provides connectivity to a different set of hosts. There is only one path that any given packet can take, and for outbound traffic this can be deduced from the destination IP address. All that is needed is a routing table that reflects which hosts are reachable through each interface.

But when there are two or more possible paths then a decision must be made as to which one to use. A condition that you should normally try to avoid is ‘asymmetric routing’, whereby traffic sent to a given IP address arrives on one interface, but traffic originating from that address leaves by a different interface. There are a number of problems that this can cause:

• Packets sent through the wrong interface will appear to have a spoofed source address, and may therefore be blocked as a matter of policy. It is common practice for Internet service providers to do this.
• Network devices which depend on connection tracking, such as stateful firewalls and intrusion detection systems, usually need to see both inbound and outbound traffic if they are to function correctly.
• Routing most outbound traffic through one interface can cause unnecessary congestion.
• If the intention was to provide a redundant path by which the server can be reached then routing all return traffic through a single interface will frustrate this.

Configuration Scenario: A Linux instance has multiple interfaces and each is connected to different network and getting the IP via DHCP, each IP as a floating IP associated to it.

• The first via ens3 with an IP address of 192.168.100.19/24 and a gateway address of 192.168.100.1; floating IP via 95.177.217.117;
• The second via ens4 with an IP address of 10.100.1.9/24 and a gateway address of 10.200.1.1; floating IP via 95.177.217.119;

The objective is the instance must be able to accept inbound TCP connections from arbitrary locations on the Internet to either of its public IP addresses and ensure that the outbound traffic associated each connection is sent through the same interface as the inbound traffic. We can verify this by ensuring all IP pingable and accessible via SSH.

To obtain such behavior we need to use the policy-based routing capabilities of the Linux kernel, which allows for routing decisions to be based on criteria other than the destination address. The method described here has four steps:

1. Ensure that the main routing table has a default route and disable it from other interfaces.
2. Create a separate routing table for each of the interfaces.
3. Add policy rules to direct outbound traffic to the appropriate routing table.
4. Flush the routing cache.

This approach can be scaled trivially to any number of interfaces if required.

Ensure that the main routing table has a default route

When a program initiates an outbound connection it is normal for it to use the wildcard source address (0.0.0.0), indicating no preference as to which interface is used provided that the relevant destination address is reachable. This is not replaced by a specific source address until after the routing decision has been made. Traffic associated with such connections will not therefore match either of the above policy rules, and will not be directed to either of the newly-added routing tables. Assuming an otherwise normal configuration, it will instead fall through to the main routing table.

The main routing table is the best place to handle this traffic because it does not require any special treatment on account of its source address. An ordinary default route via one of the available gateways will suffice:

ip route add default via 192.168.100.1

Set the first interface as default route

• Add the line “DEFROUTE=yes” (without the quote) to /etc/network/interfaces.d/50-cloud-init.cfg for interface ens3
• Add the line “DEFROUTE=no” (without the quote) to /etc/network/interfaces.d/50-cloud-init.cfg for interface ens4

Restart the network and the IP routing table should be something like this:

Above, the only pingable/reachable would be the ens3 IP only as it carries the default gateway.

Create a separate routing table for each of the interfaces

The requirement cannot be met by a single routing table, but with policy-based routing it is possible to have multiple tables. These can be created in two steps:

1. Edit the file  /etc/iproute2/rt_tables and set the ens3 with preference 1, and ens4 with 2. Any name to represent it can work, this is basically creating a policy in the routing table that can be called later.

2. Configure the routing table. Configure the 10.200.1.0/24 to tell that it can be reached in “ens4” and to find its gateway. And so on if there are more interfaces.

ip route add 10.200.1.0/24 dev ens4 src 10.200.1.9 table ens4
ip route add default via 10.200.1.1 dev ens4 table ens4

Add policy rules to direct outbound traffic to the appropriate routing table

Set the policy routes for the system to know when to use that new routing table. In our case, two policy rules are needed, one for each of the above routing tables, to arrange for the tables to be consulted when packets from the corresponding source address are seen (This should be done for all other interfaces):

ip rule add from 10.200.1.9/32 table ens4
ip rule add to 10.200.1.9/32 table ens4

Flush the routing cache

When the routing tables are queried the outcome is cached for efficiency, but according to the iproute2 documentation the cache is not flushed automatically when rules are added or removed. For this reason, the cache should be flushed explicitly once you have finished making changes:

ip route flush cache

In practice recent kernels do appear to perform an implicit cache flush, however relying on this behaviour would be very much at your own risk so long as the documentation says otherwise. Flushing twice is not harmful.

Testing and Verification

You can verify that traffic is being sent via the appropriate interface by pinging via interfaces. All the IP address see if it’s already reachable

• Rule 32764 means that all traffic going to ens4 will use the “ens4” routing table.
• Rule 32765 means that traffic coming from ens4 will use the ”ens4” routing table. And so on, and so forth

Below is the full routing table

Further reading

• ip(8)  (Ubuntu manpage)
• Linux Advanced Routing & Traffic Control HOWTO
• Matthew G. Marsh, Policy Routing With Linux
• Policy Routing (Linux kernel documentation)