Hetzner’s ban, NAT leaking internal traffic on WAN interface and New Year’s

 

As I mentioned few times, we have small cluster of servers located on Hetzner’s datacenters. We use Vyatta for our core router and we NAT all additional servers thru it, becouse Hetzner’s dedicated servers don’t support direct routing.

Incident

On new year’s eve, my phone started ringing insanely and I noticed one of our core routers is down. Checking emails I noticed Hetzner simply disabled our router server and sent email claiming our internal packets are hitting their gateways.

Part of email recieved from Hetzner:

Dear Sir or Madam

We have noticed that you have been using other IPs from the same subnet in addition to the main IP mentioned in the above subject line.

As this is not permitted, we regret to inform you that your server has been deactivated.

Guidelines regarding further course of action may be found in our wiki: http://wiki.hetzner.de/index.php/Leitfaden_bei_Serversperrung/en.

Yours faithfully

Your Hetzner Support Team

11:41:57.649487 68:05:ca:0c:ab:ff > 78:fe:3d:47:18:0d, ethertype IPv4 
(0x0800), length 74: 10.10.10.211.80 > 212.200.65.xxx.15394: Flags [S.], 
seq 1811651542, ack 933063600, win 14480, options [mss 1460,sackOK,TS val 
557219346 ecr 6976797,nop,wscale 7], length 0
11:41:57.662876 68:05:ca:0c:ab:ff > 78:fe:3d:47:18:0d, ethertype IPv4 
(0x0800), length 66: 10.10.10.211.80 > 174.140.137.xxx.63787: Flags [S.], 
seq 341019190, ack 1361019512, win 14600, options [mss 
1460,nop,nop,sackOK,nop,wscale 7], length 0
11:41:57.662881 68:05:ca:0c:ab:ff > 78:fe:3d:47:18:0d, ethertype IPv4 
(0x0800), length 74: 10.10.10.211.80 > 178.222.13.xxx.45890: Flags [S.], seq 
703546553, ack 287696831, win 14480, options [mss 1460,sackOK,TS val 
557219380 ecr 1228421,nop,wscale 7], length 0
11:41:57.670449 68:05:ca:0c:ab:ff > 78:fe:3d:47:18:0d, ethertype IPv4 
(0x0800), length 74: 10.10.10.211.80 > 109.93.4.xxx.51783: Flags [S.], seq 
481954038, ack 4142858854, win 14480, options [mss 1460,sackOK,TS val 
557219381 ecr 8775198,nop,wscale 7], length 0
11:41:57.672957 68:05:ca:0c:ab:ff > 78:fe:3d:47:18:0d, ethertype IPv4 
(0x0800), length 74: 10.10.10.211.80 > 212.200.65.xxx.59942: Flags [S.], seq 
3778211473, ack 3343731360, win 14480, options [mss 1460,sackOK,TS val 
557219382 ecr 1298478297,nop,wscale 7], length 0
11:41:57.675241 68:05:ca:0c:ab:ff > 78:fe:3d:47:18:0d, ethertype IPv4 
(0x0800), length 74: 10.10.10.211.80 > 95.87.245.xxx.61631: Flags [S.], seq 
1161393471, ack 2601691340, win 14480, options [mss 1460,sackOK,TS val 
557219383 ecr 4037672,nop,wscale 7], length 0
11:41:57.697377 68:05:ca:0c:ab:ff > 78:fe:3d:47:18:0d, ethertype IPv4 
(0x0800), length 74: 10.10.10.211.80 > 91.150.103.xxx.58613: Flags [S.], 
seq 2267229551, ack 3802744569, win 14480, options [mss 1460,sackOK,TS val 
557219388 ecr 13334862,nop,wscale 7], length 0
11:41:57.699275 68:05:ca:0c:ab:ff > 78:fe:3d:47:18:0d, ethertype IPv4 
(0x0800), length 74: 10.10.10.211.80 > 212.200.65.xxx.21635: Flags [S.], 
seq 3155739128, ack 3912318921, win 14480, options [mss 1460,sackOK,TS val 
557219389 ecr 6976502,nop,wscale 7], length 0
… And so on ...

Note: 10.10.10.211 in this case is our loadbalancer’s internal IP address and the others are clients making http requests.

At first I was surprised that is even possible and thought this is bug in Vyatta OS. Why would my router send these packets un-nated to my default gateway? Strange. I even double checked that this is actually happening, by tcpdump-ing my external interface.

Here’s link to bug report and discussion on Vyos bugtracker: http://bugzilla.vyos.net/show_bug.cgi?id=116 (Big thanks to fromport from ##vyatta channel @ Freenode for reporting this one for me.)

Discussion / Conclusion

Later on, talking with several network engineers, we came to conclusion, that this happens on other router platforms too. It happens becouse sometimes packets get dropped from connection tracking table and therefore aren’t properly NAT-ed. The more traffic you have, more likely you will have these packets hitting your gateway. Basically this happens to everyone using NAT protocol, if they don’t have firewall rule strictly forbidding this. In general such packets are (should be) silently discarded by ISP, but (unfortunately) not in our case.

Resolution

There were several options at that time:

  • Add firewall rule blocking all internal traffic on external interface and cross fingers Vyatta will block these packets
  • Setup plain debian/linux server and setup iptables to make sure it’s not Vyatta bug (in case it was a bug, no iptables firewall rule would help probably)
  • Move whole datacenter to another ISP provider

As you may guess I really didn’t want to migrate big part of our infrastructure to another ISP, just becouse Hetzner enforces such strict rules.

I tried several options and finally came to conclusion. Best way to make sure packets don’t exit external interface un-nated was to actually block all internal packets on WAN interface.

Boy, was I wrong. I forgot internal packets are used on external interface, when you use site-to-site openvpn tunnels and therefore you can’t block them.

After some time experimenting I finally found a solution. Correct way of blocking these packets are (assuming eth0 is your WAN interface):

iptables -t filter -I FORWARD -o eth0 -m conntrack --ctstate INVALID -j DROP

As soon as I solved the riddle and executed this rule, I couldn’t see internal packets on my WAN interface anymore and OpenVPN tunnels were up and running.

Bingo!

I sent email to Hetzner with request to verify this on their end too. They answered that they can not see internal packets anymore and unblocked our core router.

What a new year… :)

Credits to those who helped

Big credits for this blog post go to fromport from ##Vyatta and renihs / ard_shell3 from #netfilter channel @ Freenode. They helped me a lot by verifying my conclusions on their end and proposing solutions. 

Leave a Reply

help-hint.png
Purpose of the commenting system is to share your experience. I encourage you to post feedback with your own suggestions, ideas or optimizations regarding the topic of a blog post. What commenting system isn't for, is asking questions about similar issues of yours and requesting support for it. Blog post is provided as is and I am not here to solve all your problems. Please bear that in mind and try to avoid posting such comments. I do take privilege to remove comment from my blog for any reason whatsoever. Usually I do it when I sense a comment was posted only for spam/seo reasons or is out of blog post's topic. Thank you for reading this, now you may continue :)
 

Your email address will not be published. Required fields are marked *