Loadbalancing / failover with IPVS and keepalived

Introduction

Correct failover and loadbalancing is crucial for high availablility environment. With proper setup we can eliminate single points of failure in case of server crash. I use linux kernel’s support for load balancing, since that seems as well documented and scalable method. What I want to achieve here, is actually fully redundant architecture, so that in case one server fails, the other takes over. For such architecture, you need to have at least 4 servers or in case you want to NAT your internal servers, you need 2. I won’t go into details of how to setup NAT architecture, since it has it’s own limitations. Besides I don’t use it anywhere.

Schema

Basic topology with 4 servers would be:

Quick explanation of the above image.

• Router has simple NAT, but you can remove router and use loadbalancers with public ip’s if you have your own dedicated range of IP’s.
• Loadbalancer1 is master for ip: 10.10.10.10 which will identify our webservers behind loadbalancers.
• Loadbalancer2 is in backup state for ip: 10.10.10.10, meaning that in case Loadbalancer1 drops and stops advertising that it owns 10.10.10.10 ip thru VRRP, this router will take over the very same IP.
• Realserver1 has ip: 10.10.10.41 and dummy address 10.10.10.10
• Realserver2 has ip: 10.10.10.42 and dummy address 10.10.10.10

Loadbalancers in general

We need minimal installation of Ubuntu/CentOS linux server. We should install keepalived and ipvsadm with command apt-get install keepalived ipvsadm or yum install keepalived ipvsadm and it should install all prerequisites for running load balancer.

Loadbalancer1

We should configure our keepalived on Loadbalancer1 accordingly:

! Configuration File for keepalived

global_defs {
   notification_email {
       [email protected]
   }
   notification_email_from [email protected]
   smtp_server localhost
   smtp_connect_timeout 30
! UNIQUE:
   router_id LVS_PRI
}

! ***********************************************************************
! *************************   WEB SERVICES VIP  *************************
! ***********************************************************************
vrrp_instance VirtIP_10 {
    state MASTER
    interface eth0
    virtual_router_id 10
! UNIQUE:
    priority 150
    advert_int 3
    smtp_alert
    authentication {
        auth_type PASS
        auth_pass MY_PASS
    }
    virtual_ipaddress {
        10.10.10.10
    }

    lvs_sync_daemon_interface eth0
}

! ************************   WEB SERVERS  **************************

virtual_server 10.10.10.10 80 {
    delay_loop 10
    lvs_sched wlc
    lvs_method DR
    persistence_timeout 5
    protocol TCP

    real_server 10.10.10.41 80 {
        weight 50
        TCP_CHECK {
            connect_timeout 3
        }
    }

    real_server 10.10.10.42 80 {
        weight 50
        TCP_CHECK {
            connect_timeout 3
        }
    }

}

Loadbalancer2

For Loadbalancer2 we should set configuration accordingly:

! Configuration File for keepalived

global_defs {
   notification_email {
        [email protected]
   }
   notification_email_from [email protected]
   smtp_server localhost
   smtp_connect_timeout 30
! UNIQUE:
   router_id LVS_SEC
}

! ***********************************************************************
! *************************   WEB SERVICES VIP  *************************
! ***********************************************************************

vrrp_instance VirtIP_10 {
    state BACKUP
    interface eth0
    virtual_router_id 10
! UNIQUE:
    priority 50
    advert_int 3
    smtp_alert
    authentication {
        auth_type PASS
        auth_pass MY_PASS
    }
    virtual_ipaddress {
        10.10.10.10
    }

    lvs_sync_daemon_interface eth0
}

! ************************   WEB SERVERS  **************************

virtual_server 10.10.10.10 80 {
    delay_loop 10
    lvs_sched wlc
    lvs_method DR
    persistence_timeout 5
    protocol TCP

    real_server 10.10.10.41 80 {
        weight 50
        TCP_CHECK {
            connect_timeout 3
        }
    }

    real_server 10.10.10.42 80 {
        weight 50
        TCP_CHECK {
            connect_timeout 3
        }
    }
}

Note: All our real servers have weight 50, meaning that each real server will get half of the connections.

Tricky part / dummy addresses

If we connect to our Loadbalancer1‘s ip 10.10.10.10:80, we should be forwarded to one of our real servers. But it still doesn’t work – why is that? The tricky part in this architecture is that our real servers don’t know they actually own 10.10.10.10 ip. Therefore we need to create dummy interface on these servers. Why dummy you would ask? Becouse in LAN you can not have multiple servers advertising same ip or you will have IP address conflict. For this purpose linux knows dummy interfaces that don’t advertise their ip address in ARP.

Realserver1 and Realserver2 setup

You should bring up dummy interface on each realserver with command:

modprobe dummy numdummies=1
ifconfig dummy0 10.10.10.10 netmask 255.255.255.0

ifconfig dummy0 should return something similar to this:

dummy0 Link encap:Ethernet HWaddr F6:65:D6:D9:BB:E2
inet addr:10.10.10.10 Bcast:10.10.10.255 Mask:255.255.255.0
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

Testing setup

This is your dummy0 address, now if you try to connect from outside to your 10.10.10.10:80 , you will see you will get response from either Realserver1 or Realserver2 (I am assuming that you have already installed some sort of webserver on these two servers). You should try and stress test this infrastructure, to make sure it works.

Note: if you are using dummy addresses you should make sure you don’t announce your ip’s on the network. These settings in general go to sysctl of host server and not guest server. To fix that my /etc/sysctl.conf has these settings, yours might need a slight modification:

net.ipv4.ip_forward=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.conf.default.proxy_arp=1

net.ipv4.conf.all.promote_secondaries=1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

net.ipv4.conf.virbr0.arp_ignore = 1
net.ipv4.conf.virbr0.arp_announce = 2
net.ipv4.conf.eth1.arp_announce = 0

Notes:

• eth1 is internal interface of host, I made virbr0 on it.
• virbr0 is my bridge on eth1 interface – yours can be vmnet, xenbr or something similar. It’s basically bridge interface on which you spawn your virtual servers.