Tips for optimizing disk performance on Linux

Forespeech

Business Performance GraphLately I’ve been dealing mostly with storage and disk performance. We have several MySQL / MongoDB databases, which can be very I/O consuming. Usually people benchmark storage when they setup server, but running a real service on it, will actually show how really optimized the server is.

Tips for optimizing your disk storage

I’ll give you few tips for optimizing your storage and what to check. They are not all the aspects I think of, when setting up a storage, but they sure are the major ones.

  • Disk Cache Policy

Be sure to enable this one on your RAID controller. It’s usually On by default, but it’s worth to check. You should also check that your Linux actually uses disk cache policy by executing: hdparm -W /dev/sda , also check: hdparm -a /dev/sda to see readahead value. It might be useful to set this parameter to at least 1024. Test different readahead values by testing read performance with dd and choose best one that fits your storage.

  • Write Policy on RAID Controller

Very important thing to check. Be sure it’s writeback for best performance. It uses BBU unit to make sure if server crashes or there’s power outage it will still flush data on disks.

  • Read Policy on RAID Controller

I usually keep it Adaptive readahead. Some controllers allow you to set this, some don’t. Set it if you have it.

  • Raid Level

Check Raid’s level write penalties before you choose it. It’s very important aspect for performance.

penalties

  • IOPS of your disks

Check maximum input/output operations per second of your disk. For databases it’s most important aspect. It’s all about IOPS.

http://en.wikipedia.org/wiki/IOPS

Example calculating Max IOPS for raid 10 with 8 disks:

SAS 15k rpm disk: 175-210 IOPS ~ 200 IOPS

200 * 8 = 1600 IOPS for only read performance
200 * 8 / 2 = 800 IOPS for only write performance (becouse of write performance penalty).

Explaination: Raid 10 with 8 disks consists of 4 raid 1 arrays connected with raid 0. Raid 1 has to write on both disks and raid 0 just writes on one, meaning you have to write on 2 disks for one write. Therefore write penalty of 2. Here’s Raid 10 architecture:

raid-10

  • Stripe Element Size

You should usually set this to 65KB or 128KB max.

  • Filesystem type

For databases you should use xfs or at least ext4. Filesystem optimized for your usecase can help a lot.

  • I/O Scheduler

There are several I/O schedulers availiable in Linux. Default one is cfq. Even if you think this is not important aspect, I will try to convince you otherwise.

We have write heavy database server, that’s totally draining our storage. 100% utilization of disks and large await for read/write requests. I was testing Linux I/O schedulers the other day and came to great results. First deadline (at around 9-11 o’clock), which showed minor improovement and afterwards noop (at 11:30 o’clock). You can see the difference when I changed scheduler to noop and left it there on these graphs:

util_elephant

elephant
 
To check availiable Linux I/O schedulers: cat /sys/block/sda/queue/scheduler
To set certain Linux I/O scheduler: echo noop > /sys/block/sda/queue/scheduler

Final words

I optimized this storage a little and we prolonged the time needed to buy a new one. You are very welcome to comment under this post and tell everyone your own ways of improoving performance!

Leave a Reply

help-hint.png
Purpose of the commenting system is to share your experience. I encourage you to post feedback with your own suggestions, ideas or optimizations regarding the topic of a blog post. What commenting system isn't for, is asking questions about similar issues of yours and requesting support for it. Blog post is provided as is and I am not here to solve all your problems. Please bear that in mind and try to avoid posting such comments. I do take privilege to remove comment from my blog for any reason whatsoever. Usually I do it when I sense a comment was posted only for spam/seo reasons or is out of blog post's topic. Thank you for reading this, now you may continue :)
 

Your email address will not be published. Required fields are marked *