Loadbalancing / failover with IPVS and keepalived
Introduction
Correct failover and loadbalancing is crucial for high availablility environment. With proper setup we can eliminate single points of failure in case of server crash. I use linux kernel’s support for load balancing, since that seems as well documented and scalable method. What I want to achieve here, is actually fully redundant architecture, so that in case one server fails, the other takes over. For such architecture, you need to have at least 4 servers or in case you want to NAT your internal servers, you need 2. I won’t go into details of how to setup NAT architecture, since it has it’s own limitations. Besides I don’t use it anywhere.
Schema
Basic topology with 4 servers would be:
Quick explanation of the above image.
- Router has simple NAT, but you can remove router and use loadbalancers with public ip’s if you have your own dedicated range of IP’s.
- Loadbalancer1 is master for ip: 10.10.10.10 which will identify our webservers behind loadbalancers.
- Loadbalancer2 is in backup state for ip: 10.10.10.10, meaning that in case Loadbalancer1 drops and stops advertising that it owns 10.10.10.10 ip thru VRRP, this router will take over the very same IP.
- Realserver1 has ip: 10.10.10.41 and dummy address 10.10.10.10
- Realserver2 has ip: 10.10.10.42 and dummy address 10.10.10.10
Loadbalancers in general
We need minimal installation of Ubuntu/CentOS linux server. We should install keepalived and ipvsadm with command apt-get install keepalived ipvsadm
or yum install keepalived ipvsadm
and it should install all prerequisites for running load balancer.
Loadbalancer1
We should configure our keepalived on Loadbalancer1 accordingly:
! Configuration File for keepalived global_defs { notification_email { [email protected] } notification_email_from [email protected] smtp_server localhost smtp_connect_timeout 30 ! UNIQUE: router_id LVS_PRI } ! *********************************************************************** ! ************************* WEB SERVICES VIP ************************* ! *********************************************************************** vrrp_instance VirtIP_10 { state MASTER interface eth0 virtual_router_id 10 ! UNIQUE: priority 150 advert_int 3 smtp_alert authentication { auth_type PASS auth_pass MY_PASS } virtual_ipaddress { 10.10.10.10 } lvs_sync_daemon_interface eth0 } ! ************************ WEB SERVERS ************************** virtual_server 10.10.10.10 80 { delay_loop 10 lvs_sched wlc lvs_method DR persistence_timeout 5 protocol TCP real_server 10.10.10.41 80 { weight 50 TCP_CHECK { connect_timeout 3 } } real_server 10.10.10.42 80 { weight 50 TCP_CHECK { connect_timeout 3 } } }
Loadbalancer2
For Loadbalancer2 we should set configuration accordingly:
! Configuration File for keepalived global_defs { notification_email { [email protected] } notification_email_from [email protected] smtp_server localhost smtp_connect_timeout 30 ! UNIQUE: router_id LVS_SEC } ! *********************************************************************** ! ************************* WEB SERVICES VIP ************************* ! *********************************************************************** vrrp_instance VirtIP_10 { state BACKUP interface eth0 virtual_router_id 10 ! UNIQUE: priority 50 advert_int 3 smtp_alert authentication { auth_type PASS auth_pass MY_PASS } virtual_ipaddress { 10.10.10.10 } lvs_sync_daemon_interface eth0 } ! ************************ WEB SERVERS ************************** virtual_server 10.10.10.10 80 { delay_loop 10 lvs_sched wlc lvs_method DR persistence_timeout 5 protocol TCP real_server 10.10.10.41 80 { weight 50 TCP_CHECK { connect_timeout 3 } } real_server 10.10.10.42 80 { weight 50 TCP_CHECK { connect_timeout 3 } } }
Note: All our real servers have weight 50, meaning that each real server will get half of the connections.
Tricky part / dummy addresses
If we connect to our Loadbalancer1‘s ip 10.10.10.10:80, we should be forwarded to one of our real servers. But it still doesn’t work – why is that? The tricky part in this architecture is that our real servers don’t know they actually own 10.10.10.10 ip. Therefore we need to create dummy interface on these servers. Why dummy you would ask? Becouse in LAN you can not have multiple servers advertising same ip or you will have IP address conflict. For this purpose linux knows dummy interfaces that don’t advertise their ip address in ARP.
Realserver1 and Realserver2 setup
You should bring up dummy interface on each realserver with command:
modprobe dummy numdummies=1 ifconfig dummy0 10.10.10.10 netmask 255.255.255.0
ifconfig dummy0
should return something similar to this:
dummy0 Link encap:Ethernet HWaddr F6:65:D6:D9:BB:E2 inet addr:10.10.10.10 Bcast:10.10.10.255 Mask:255.255.255.0 UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Testing setup
This is your dummy0 address, now if you try to connect from outside to your 10.10.10.10:80 , you will see you will get response from either Realserver1 or Realserver2 (I am assuming that you have already installed some sort of webserver on these two servers). You should try and stress test this infrastructure, to make sure it works.
Note: if you are using dummy addresses you should make sure you don’t announce your ip’s on the network. These settings in general go to sysctl of host server and not guest server. To fix that my /etc/sysctl.conf
has these settings, yours might need a slight modification:
net.ipv4.ip_forward=1 net.ipv4.conf.all.rp_filter=1 net.ipv4.icmp_echo_ignore_broadcasts=1 net.ipv4.conf.default.proxy_arp=1 net.ipv4.conf.all.promote_secondaries=1 net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.virbr0.arp_ignore = 1 net.ipv4.conf.virbr0.arp_announce = 2 net.ipv4.conf.eth1.arp_announce = 0
Notes:
- eth1 is internal interface of host, I made virbr0 on it.
- virbr0 is my bridge on eth1 interface – yours can be vmnet, xenbr or something similar. It’s basically bridge interface on which you spawn your virtual servers.

Yoooo Alen… Great read bro! keep it up
Thanks!
Hello Alen, thanks for your post,
Small questions :
– 1
I’m asking why you use dummy0 and not lo:0 ?
If you use : ifconfig lo:1 10.10.10.10 -arp netmask 255.255.255.0 up
don’t you have the same thing ?
-2
Why are you talking about : eth1 and virbr0 ?
In a simple topology, real server use eth0 that’s all, not ?
-3
Could you explain for real servers your config :
net.ipv4.ip_forward=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.conf.default.proxy_arp=1
net.ipv4.conf.all.promote_secondaries=1
Thanks for your time for your reply.
Hello Ed!
First of all, thanks for asking, that’s why this blog is here.
1. As long as you take care of NOARP on the interface, it should be fine. Downside of using loopback interface is also, you can’t apply any firewall rules to it. You will run into problems. Dummy interfaces are more generic, since loopback should be used only for bouncing packets for local delivery.
2. If you use loadbalancing you probably use virtualization also. (If you wanted to test my guide you would probably want to test it in virtualization, before going with it in production environment.) It’s not really necessary, but still… in larger environments you can’t expect to run only one loadbalancer on a really strong server. It would be waste, since loadbalancer itself doesn’t really need almost any resources at all. If you use virtualization you usually bridge your virtual servers on your internal interface, meaning you use bridge interface which is in this case called virbr. They can be called otherwise too, for example Xen uses xenbr0, VMWare has vmnet0 etc… Don’t mix host’s bridge interface, and guests’s interface example: eth0. Real servers don’t need any modification other than dummy interface.
3. These sysctl rules are on the host of virtual servers and not guest (real server). (Maybe I should edit my post to make that more clear…)
– proxy_arp and ip_forward control IP packet forwarding to your guests, therefore you need it if you want them to be able to see each other and have internet access of host.
– rp_filter is by default included in sysctl as far as I know and it’s security feature. You don’t want packets originating from elsewhere than your network (for example 192.168.0.0/24) to be routed thru your network interfaces. This setting prevents that.
– I like to ignore icmp broadcasts, since that can make problems in general, if you have larger number of servers. Besides noone should be pinging broadcast address, unless you are hacker and you want to get list of active ip’s on the network with one command. (this line can be ignored if you don’t want it, but I suggest you to leave it.)
– As of promote_secondaries , it’s more feature of administration, than necessity. If you have more ip’s on your interface and want to change primary ip, you would loose all secondary ip’s also. Basically meaning, you would be left locked out of server and would have to restart it, then pray your old config still has primary ip set and would be brought back. You must understand these settings were taken from working environment and some of them are there becouse they were needed in the long run, I wanted to save you some time figuring it out on your own.
Thanks for your clear reply.
Dummy will be able to help me in my topology so – with the help of iptables perhaps.
Behind the Load Balancer, I have 2 Real Servers (web + database) :
A = First one : web + database master
B = Second one : web + database slave (replication)
I don’t succeed to make that database requests (from web sites on A and B) go to load balancer (that distribute the database charge) not on localhost (actually the requests are made on the real server directly)… and I think it’s because lo:1 has the database virtual IP and i couldn’t hide it from localhost.
For information in my case lo:0 has the web virtual IP on each real server.
Of course my sysctl.conf has : net.ipv4.conf.all.arp_ignore = 1 and net.ipv4.conf.all.arp_announce = 2
Do you think i’am on the right way, change “lo” by “dummy” (and perhaps iptables rules) ?
Thanks in adavance.
I’m afraid dummy interface won’t fix your problem. In basic, host that has dummy IP address, will always want to deliver it on that interface, no way to avoid that. Meaning if you want to do loadbalancing with only 2 physical servers, it’s not possible. Or at least I haven’t figured out how to do it. You should put databases on separate physical servers, other than frontend servers, then loadbalancing is possible and works perfectly. Otherwise, no. Also be careful when balancing database servers, since you want to make sure writes go on one server only. Not to make collision. You can weight your server for that matter.
Great Tutorial! Loadbalancing workfs fine! But careful with the dumy interface… after a reboot it is gone. Put the commands in autostart to keep it working!