Lack of linux kernel entropy – Example how things can go wrong

The issue

Lately we’ve been experimenting with puppet and setting up our frontend webservers with it. Our junior sysadmin was setting up things on a server that already had frontend database virtualized and everything was going smoothly, to the part, when he had to setup whole webserver from scratch.

I was working with Percona’s XtraDB MySQL Cluster in my development server, when I noticed serious drops on our frontend server requests. Our frontend started to “saw” like this (example graph from our frontend webserver):

lighty_sawing

First thought was, that we synced some new code to our webservers and it’s broken. I couldn’t be more wrong. I suspected one of our databases suddenly became really slow. Junior sysadmin mentioned that he’s running puppet next to one of them and things got a bit clearer.

Investigation

When I checked graphs for the host I couldn’t find real issue. Sure, some I/O spikes and minor CPU usage, but this isn’t real problem. This server is totally idle and should work without any problem. Looking further I came across the entropy graph. It looked like this:

entropy_spikes

I realised this may be the cause for such problems.

Conclusion

When I installed a neat program called haveged, our graph turned totally around. I told junior sysadmin to start his puppet again to see if issues with frontend persist. No spike whatsoever on the frontend. Everything working as should. Looking at the entropy graph showed this: 

enough_entropy

I couldn’t believe that lack of entropy on one of our servers, could have so dramatic effect on our webserving. I’m pretty happy to have solved this issue which is pretty hard to trace and debug, especially if you don’t know what’s going on.

Leave a Reply

help-hint.png
Purpose of the commenting system is to share your experience. I encourage you to post feedback with your own suggestions, ideas or optimizations regarding the topic of a blog post. What commenting system isn't for, is asking questions about similar issues of yours and requesting support for it. Blog post is provided as is and I am not here to solve all your problems. Please bear that in mind and try to avoid posting such comments. I do take privilege to remove comment from my blog for any reason whatsoever. Usually I do it when I sense a comment was posted only for spam/seo reasons or is out of blog post's topic. Thank you for reading this, now you may continue :)
 

Your email address will not be published. Required fields are marked *