Lately we’ve been experimenting with puppet and setting up our frontend webservers with it. Our junior sysadmin was setting up things on a server that already had frontend database virtualized and everything was going smoothly, to the part, when he had to setup whole webserver from scratch.
I was working with Percona’s XtraDB MySQL Cluster in my development server, when I noticed serious drops on our frontend server requests. Our frontend started to “saw” like this (example graph from our frontend webserver):
First thought was, that we synced some new code to our webservers and it’s broken. I couldn’t be more wrong. I suspected one of our databases suddenly became really slow. Junior sysadmin mentioned that he’s running puppet next to one of them and things got a bit clearer.
When I checked graphs for the host I couldn’t find real issue. Sure, some I/O spikes and minor CPU usage, but this isn’t real problem. This server is totally idle and should work without any problem. Looking further I came across the entropy graph. It looked like this:
I realised this may be the cause for such problems.
When I installed a neat program called haveged, our graph turned totally around. I told junior sysadmin to start his puppet again to see if issues with frontend persist. No spike whatsoever on the frontend. Everything working as should. Looking at the entropy graph showed this:
I couldn’t believe that lack of entropy on one of our servers, could have so dramatic effect on our webserving. I’m pretty happy to have solved this issue which is pretty hard to trace and debug, especially if you don’t know what’s going on.