Polipo: Easy Transparent proxy

When your server numbers grow and you have clusters you want to upgrade all at one time, you will probably want some kind of proxy that will cache stuff for you. Upgrading cluster of hadoop with like 10 servers, can take up to 2hours just to download packages to all nodes, since Cloudera limits bandwidth to 125kB/s per download. This can be pretty painful without proxy and your downtime would increase a lot. There are several other occasions where you can benefit from proxy:

• upgrading distributions
• updating your servers to latest packages
• downloading packages with yumdownloader etc.

To speed things up, we need really small and easy to setup proxy with caching ability. Introducing Polipo. Unlike Squid which is total pain to configure and set up, Polipo is quick and easy. I got it running in under 10 minutes (i wanted to read all config file), but you can simply copy my configuration and get it up straight a way.

Installing polipo

Polipo is included in Epel 6 repository. To get it, we first need to install Epel repository and then install Polipo:

rpm -Uvh http://mirror.uv.es/mirror/fedora-epel/6/i386/epel-release-6-8.noarch.rpm
yum install polipo

Configuring polipo

Grab following config file and copy it into /etc/polipo/config Be sure to edit:

• allowedClients to your server’s network range
• proxyAddress (0.0.0.0 would listen to all interfaces)
• dnsQueryIPv6 in case you use IPv6

daemonise = true
pidFile = /var/run/polipo/polipo.pid
proxyAddress = "0.0.0.0"
allowedClients ="xxx.xxx.xxx.xxx/xx"
cacheIsShared = true
chunkHighMark = 50331648
objectHighMark = 16384
dnsQueryIPv6 = no

We should simply start our web proxy now with: service polipo start

To make sure our yum starts using our proxy, we should add a line to /etc/yum.conf on all client servers we want to use with this proxy server:

proxy=http://hostname_or_ip_of_your_polipo_server:8123

To test your proxy works perfectly, try downloading some large file with yumdownloader for example: yumdownloader libreoffice-core First time it actually downloads package from repository, but if you repeat this command on another server, download should start directly from proxy server and therefore really fast.