Environment

Ubuntu-18.04-LTS with two FIPs, connected directly to an external router.

• First Interface information:

                 Name: ens3
                 IP address: 10.20.30.8/24
                 GW: 10.20.30.1
                 MAC address: 02:2f:79:d0:16:90

• Second Interface information:

                 Name: ens4
                 IP address: 10.30.30.4/24
                 GW: 10.30.30.1
                 MAC adress: 02:88:75:91:9f:a2

Steps

1- Update and Upgrade the list of available packages inside the system: sudo apt update sudo apt upgrade

2- Add the default route to be on first interface: ip route add default via 10.20.30.1

3- Edit the file /etc/netplan/50-cloud-init.yaml with the following content:

# This file is generated from information provided by
# the datasource.  Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        ens3:
            dhcp4: true
            dhcp4-overrides:
                use-routes: true
            match:
                macaddress: 02:2f:79:d0:16:90
            set-name: ens3

        ens4:
            dhcp4: true
            dhcp4-overrides:
                use-routes: false
            match:
                macaddress: 02:88:75:91:9f:a2
            set-name: ens4

4- Applying the following two commands: netplan generate netplan apply

5- Edit the file /etc/iproute2/rt_tables and set the ens3 with preference 1, and ens4 with 2:

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
1 ens3
2 ens4

6- Create an rc.local file on path /etc/rc.local with the following content:

#!/bin/bash
ip route add 10.30.30.0/24 dev ens4 src 10.30.30.4 table ens4
ip route add default via 10.30.30.1 dev ens4 table ens4
ip rule add from 10.30.30.4/32 table ens4
ip rule add to 10.30.30.4/32 table ens4
ip route flush cache

7- You may need to modify the permissions of the /etc/rc.local file by using: chmod 755 /etc/rc.local

8- Restart rc.local service: systemctl restart rc.local.service