Vyatta – SNAT – Randomly rotating public IP addresses

vyatta_logo
The problem

Our contextual system uses bots/spiders to leech data from our customer’s websites in order to parse them and use extracted text for further content analysis. This means we have setup few servers that run bots/spiders. Can you imagine if 20 bots started leeching websites hosted on same webhosting using only one ip? That could be considered as sort of a DOS attack. Hosting providers tend to block offending ip’s that connect too many times and we want to avoid that. Answer lays in ip rotation. We can use a set of ip addresses for these bots and our data leech from one ip would be reduced by a number of ip’s added to rotation pool.

The solution

Solution is hidden directly in Vyatta. Some people use advanced iptables rules to achieve such thing, while Vyatta handles it perfectly by default. In order to rotate our outside ip’s to a set of internal ip’s, we had to setup config on our Vyatta looking similar to this:

nat {
     source {
         rule 50 {
             description "Gateway for Bots"
             outbound-interface eth0
             protocol all
             source {
                 address 10.10.10.150-10.10.10.155
             }
             translation {
                 address 193.200.66.138-193.200.66.142
             }
         }
     }
}

Our bots as you can see are located on 10.10.10.150 to 10.10.10.155 local ip addresses and they have corresponding outside ip’s from 193.200.66.138 to 193.200.66.142. eth0 in this case is external interface with public ip’s.

Testing

As we have set this nat rule up, we can test it by simply wget-ing an url over and over again. A best way to achieve that is to leech url that returns us our ip address. To do that, we just have to execute this command:

wget -q -O - checkip.dyndns.org|sed -e 's/.*Current IP Address: //' -e 's/<.*$//'

This one-liner actually returns us ip address from which we are originating from. So, now we just have to make a simple script and launch the same command like 100 times, to see if our outside ip’s actually change:

bot002 ~ $ bash whatismyip_script.sh
193.200.66.142
193.200.66.141
193.200.66.139
193.200.66.138
193.200.66.142
193.200.66.141
193.200.66.141
‚Äč...

As you can see, our IP’s are rotated randomly. Some more, some less (this was tested on live system so leeching is going on in background and some ip’s are already being used) but in general % of each used ip should be even in the long run.

Plans for future

We are planning to use this very same ip rotation for our email notification system in future. Our admins sometimes want to notify all our clients about certain changes in the system or something similar, and we want to prevent our public ip from being published on spam blacklists, for sending larger amounts of emails from one ip.

Leave a Reply

help-hint.png
Purpose of the commenting system is to share your experience. I encourage you to post feedback with your own suggestions, ideas or optimizations regarding the topic of a blog post. What commenting system isn't for, is asking questions about similar issues of yours and requesting support for it. Blog post is provided as is and I am not here to solve all your problems. Please bear that in mind and try to avoid posting such comments. I do take privilege to remove comment from my blog for any reason whatsoever. Usually I do it when I sense a comment was posted only for spam/seo reasons or is out of blog post's topic. Thank you for reading this, now you may continue :)
 

Your email address will not be published. Required fields are marked *