ip_conntrack: table full, dropping packet / conclusions about connection tracking

The problem

Recently one of our clients started a very large campaign and my servers got hit by twice the traffic it was handling normally. We actually didn’t have any problems, but from time to time you would notice this error message on our main firewall:

ip_conntrack: table full, dropping packet
Investigation

I have seen this error before and I knew I had to increase the connection tracking table. But then again, as I recall this table was already increased recently, how come it needs to be increased again? So I went investigating. Check my current limit:

router1:/home/vyatta# sysctl net.ipv4.netfilter.ip_conntrack_max
net.ipv4.netfilter.ip_conntrack_max = 917504

Okay that seems reasonably large to handle all my connections. Let’s check current count:

router1:/home/vyatta# sysctl net.ipv4.netfilter.ip_conntrack_count
net.ipv4.netfilter.ip_conntrack_count = 917115

Woops?! Okay, I thought, there must be some DOS or DDOS coming up, let’s dump our conntrack table to a file and investigate:

conntrack -L > table.txt

I started few command to drill down ip’s by destination port, source port, count connections, but couldn’t find anything special. Few blacklisted ip’s on public spam blacklists doing a bit more connections than “normal” clients, but that wasn’t such a problem so I went and checked actual packets. tail table.txt showed me some interesting lines like this:

tcp      6 370429 ESTABLISHED src=91.209.xxx.xxx dst=77.85.xxx.xxx sport=80 dport=11805 packets=1 bytes=40 [UNREPLIED] src=77.85.xxx.xxx dst=91.209.xxx.xxx sport=11805 dport=80 packets=0 bytes=0 mark=0 secmark=0 use=2

What really bothered me was, almost all this table is full of ESTABLISHED connections, but 90% have UNREPLIED and size of 40 bytes in them. I went investigating what could be cause of this and I actually found this link: http://www.netfilter.org/documentation/FAQ/netfilter-faq-3.html#ss3.17 

Quote from netfilter’s website:

The answer is easy: UNREPLIED entries are temporary entries, i.e. as soon as we run out of connection tracking entries (we reach /proc/sys/net/ipv4/ip_conntrack_max), we delete old UNREPLIED entries. In other words: instead of having empty conntrack entries, we’d rather keep some maybe useful information in them until we really need them.

Conclusion / Solution

Okay, so basically all these UNREPLIED lines were actually rubbish. Seems like this conntrack table act much like Linux memory. Linux by default uses all memory to cache as much as possible and purge unnecessary memory on demand, if active program needs it.

Okay so if we wanted we could simply purge some of these temporary entries to clean our table. This could be achieved by execing few commands:

cat table.txt | grep UNREPLIED | grep ESTABLISHED | grep "packets=1" |grep "packets=0" > unreplied.txt
cat unreplied.txt | sed 's/=/ /g'| awk '{system("echo -D -s "$6" -d "$8" -p "$1" --sport="$10" --dport="$12)}'
! Update !

What I also found out was, lowering timeout of established tcp connections affect these unreplied connections as well as generic tcp timeout. Lowering default values 600 seconds for nf_conntrack_generic_timeout and 432000 for ip_conntrack_tcp_timeout_established got us like half less data in connection tracking table. We had no more issues with hitting limits anymore.

Settings that we used:

sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_established=54000
sysctl -w net.netfilter.nf_conntrack_generic_timeout=120

Comments

  1. By Wayne

    Reply

  2. By Alen

    Reply

  3. By Axel C

    Reply

  4. Reply

    • By Axel C

      Reply

Leave a Reply

help-hint.png
Purpose of the commenting system is to share your experience. I encourage you to post feedback with your own suggestions, ideas or optimizations regarding the topic of a blog post. What commenting system isn't for, is asking questions about similar issues of yours and requesting support for it. Blog post is provided as is and I am not here to solve all your problems. Please bear that in mind and try to avoid posting such comments. I do take privilege to remove comment from my blog for any reason whatsoever. Usually I do it when I sense a comment was posted only for spam/seo reasons or is out of blog post's topic. Thank you for reading this, now you may continue :)
 

Your email address will not be published. Required fields are marked *