Howto: Libvirt, Vyatta, Failover IP and Hetzner


Overview

We will use 2 Vyatta routers, make one primary gateway for all internal servers and when primary Vyatta crashes, the backup router will take gateway IP with help of VRRP and aditionally switch frontend Failover IP from one server to another.

Prerequisites
  • 2 physical servers on Hetzner
  • Failover IP
  • Internal NIC in both servers, to route all internal servers thru the same gateway
  • Setup bridge on internal NIC, so you can bridge virtual servers to it. Mine is called virbr0.
Asumptions
  • I assume you already installed libvirt on both servers
  • I assume you have knowledge how to install Vyatta as guest using libvirt
Limitations

Hetzner forces you to use external NIC’s MAC for failover and main IP. We can not bridge our Vyatta to external NIC, becouse Hetzner will see virtualized MAC from inside the guest and you won’t be able to reach internet with the guest. In order to circumvent that, we need to passthru whole ethernet device to our Vyatta router.

To do that, we need to modify our boot and enable Intel-VT. We should change /boot/grub/menu.lst something similar to this. Pay attention to intel_iommu=on:

default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-279.el6.x86_64)
 root (hd0,0)
 kernel /vmlinuz-2.6.32-279.el6.x86_64 ro root=/dev/mapper/vg0-lv_root intel_iommu=on rd_NO_LUKS rd_LVM_LV=vg0/lv_root rd_NO_MD crashkernel=auto rd_LVM_LV=vg0/lv_swap rd_NO_DM rhgb quiet console=ttyS0
initrd /initramfs-2.6.32-279.el6.x86_64.img

Be sure to reboot your server after doing that.

Identifying pci devices
vs041 ~ $ virsh nodedev-list --cap pci
pci_0000_00_00_0
pci_0000_00_01_0
pci_0000_00_02_0
pci_0000_00_14_0
pci_0000_00_16_0
pci_0000_00_1a_0
pci_0000_00_1c_0
pci_0000_00_1c_4
pci_0000_00_1c_5
pci_0000_00_1d_0
pci_0000_00_1f_0
pci_0000_00_1f_2
pci_0000_00_1f_3
pci_0000_03_00_0
pci_0000_04_00_0

When identifying Ethernet devices, be sure to notice this 03:00.0 in front of our eth0 (external NIC):

vs041 ~ $ lspci | grep Ethernet
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 09)
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)

Now that we identified which pci device we want to add to our libvirt guest, we run:

vs041 ~ $ virsh nodedev-dumpxml pci_0000_03_00_0
<device>
  <name>pci_0000_03_00_0</name>
  <parent>pci_0000_00_1c_4</parent>
  <driver>
    <name>r8169</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>3</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x8168'>RTL8111/8168B PCI Express Gigabit Ethernet controller</product>
    <vendor id='0x10ec'>Realtek Semiconductor Co., Ltd.</vendor>
    <capability type='virt_functions'>
    </capability>
  </capability>
</device>
Adding primary NIC to Vyatta guest

We should write down: domain, bus, slot and function attributes, so we can add them to our vyatta01.xml config:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
      <address domain='0x0' bus='0x3' slot='0x0' function='0x0'/>
      </source>
    </hostdev>
Adding internal NIC

We also want internal NIC for our guest Vyatta, so we can have VRRP and default gateway on it. We can bridge this one, since there are no limitations on Hetzner regarding internal NIC’s.

    <interface type='bridge'>
      <mac address='00:f7:80:ff:ff:ff'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
    </interface>
Configuring Vyatta routers

We should boot up our Vyatta with above xml configuration and look into configuring it from CLI.

Primary

My primary Vyatta router’s config looks like this:

interfaces {
     ethernet eth0 {
         address 85.10.xxx.xxx/27
         address 88.198.xxx.xxx/32
         duplex auto
         hw-id c8:60:00:fc:39:f7
         ip {
         }
         smp_affinity auto
         speed auto
     }
     ethernet eth1 {
         address 10.1.1.5/24
         duplex auto
         hw-id 00:f7:80:ff:ff:ff
         ip {
         }
         smp_affinity auto
         speed auto
         vrrp {
             vrrp-group 1 {
                 advertise-interval 5
                 authentication {
                     password mypassword
                     type plaintext-password
                 }
                 description "VRRP gateway"
                 preempt true
                 priority 180
                 run-transition-scripts {
                     master /config/scripts/failover_ip.script
                 }
                 sync-group SYNC_GROUP1
                 virtual-address 10.1.1.1
             }
         }
     }

Description of above config

  • I have setup eth0 with mac address c8:60:00:fc:39:f7, this is MAC Hetzner provided for my external NIC, that I passthrued to Vyatta in order to use it in virtualized guest.
  • I have added another IP and this is my Failover IP I bought from Hetzner: 88.198.xxx.xxx/32
  • My second part consists of gateway address for all my internal servers. I have added IP address for my Vyatta 10.1.1.5 (Just for purpose of communication between primary and secondary router) and this is the bridged virtual interface to virbr0 on which I bridge all my internal servers.
  • I have also added VRRP IP 10.1.1.1 which is my default gateway. I advertise my IP on the network every 5 seconds and becouse I want this to be my primary router, I have set priority of 180.
  • The tricky part is run-transition-scripts. This is the script that actually does all the work of connecting to Hetzner’s API and switching the outside failover IP from one server to another.
Secondary

Once we have this, we need to setup another Vyatta on the other server using exactly the same procedure. This time config of secondary Vyatta looks like this:

 ethernet eth0 {
     address 85.10.yyy.yyy/27
     address 88.198.xxx.xxx/32
     duplex auto
     hw-id c8:60:00:dd:a2:29
     ip {
     }
     smp_affinity auto
     speed auto
 }
 ethernet eth1 {
     address 10.1.1.6/24
     duplex auto
     hw-id 00:10:aa:bb:ff:1e
     smp_affinity auto
     speed auto
     vrrp {
         vrrp-group 1 {
             advertise-interval 5
             authentication {
                 password mypassword
                 type plaintext-password
             }
             description "VRRP gateway"
             preempt true
             priority 40
             run-transition-scripts {
                 master /config/scripts/failover_ip.script
             }
             sync-group SYNC_GROUP1
             virtual-address 10.1.1.1
         }
     }
 }

Description of above config

  • I have setup eth0 on the backup Vyatta just the same as my primary. My failover IP, that I added in this case, isn’t used and therefore in dummy mode, so when Hetzner’s API switches MAC from one server to another, it will start working automatically.
  • My internal interface only has 2 differences, one is priority, and the other is IP of backup router. I want this to be my backup router so I have set priority to less than 180 which is 40 and different IP for the router itself.

You probably won’t be able to commit these configs on your Vyatta routers in case you tried, becouse you still don’t have the failover script which is: /config/scripts/failover_ip.script

Failover script

Let’s have a look at it. On Primary Vyatta I have it like this:

#!/bin/bash
curl -u username:password https://robot-ws.your-server.de/failover/88.198.xxx.xxx -d active_server_ip=85.10.xxx.xxx
/home/vyatta/sendEmail-v1.56/sendEmail -s email.server.ip -t [email protected] -f [email protected] -m "Switching outside ip to primary" -u "Vyatta: Calling script to switch failover IP to Primary"

Obviously username and password are provided to you from Hetzner and you should replace them, same goes for failover IP.

On Secondary Vyatta you can have the same script, but replace that active_server_ip to the secondary’s server external ip, replace words “primary” with “secondary” and replace “router01” with “router02”, so you will know from which router the email came.

As far as I can tell, curl is already installed on latest Vyatta router, so this should not be the problem. The second part is sendmail-v1.56 which actually consists of simple script to send email from bash. You can download it from here: Click!
Put it in your /home/vyatta/ folder and it test it manually, to make sure it works.

Testing our setup

Once we have things setup, we should test it. Check which vyatta currently owns your gateway IP – 10.1.1.1

Type ip addr list and scroll down to eth1 and verify:

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:f7:80:ff:ff:ff brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.5/24 brd 10.1.1.255 scope global eth1
    inet 10.1.1.1/32 scope global eth1
    inet6 fe80::217:3eff:fe1b:bb4d/64 scope link
       valid_lft forever preferred_lft forever

You can see your primary Vyatta owns 10.1.1.1 gateway IP.

You should crash this Vyatta and check if the other one switched the IP. At the same time as second Vyatta gets the gateway IP, it will also call the script to switch failover IP. Verify it by constantly pinging your failover ip from outside, while crashing one router, then the other.

It may take up to half a minute for your pings to come back, becouse Hetzner’s API takes up to 30 seconds to switch the IP’s mac from one to another.

I hope this quick tutorial helped you to get at least an idea of how to setup failover on Hetzner, before you rush into coding some uber complicated scripts that will check whether your server is up or not.

Comments

  1. By K Elboshra

    Reply

Leave a Reply

help-hint.png
Purpose of the commenting system is to share your experience. I encourage you to post feedback with your own suggestions, ideas or optimizations regarding the topic of a blog post. What commenting system isn't for, is asking questions about similar issues of yours and requesting support for it. Blog post is provided as is and I am not here to solve all your problems. Please bear that in mind and try to avoid posting such comments. I do take privilege to remove comment from my blog for any reason whatsoever. Usually I do it when I sense a comment was posted only for spam/seo reasons or is out of blog post's topic. Thank you for reading this, now you may continue :)
 

Your email address will not be published. Required fields are marked *