Tracing I/O usage in Linux

Introduction

We know quite few programs that help us identify I/O usage in Linux. In order to identify exactly what is writing and which process, we need to use quite few commands, to make sure we know exactly what is going on. I will show you few useful commands that will help you find your process and files that are being written from top level to the actual file.

Find disk / LVM

Identify logical volume / disk that’s causing most disk usage:

$> iostat -dx -N -m 1
Linux 4.2.0-30-generic (ibex)   03/16/2016      _x86_64_        (4 CPU)
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb               0.00     0.00    0.00    1.00     0.00     0.00     1.00     0.00    4.00   4.00   0.40
sda               0.00     0.00    0.00    1.00     0.00     0.00     1.00     0.01    5.00   5.00   0.50
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde               0.00   388.00   59.00  315.00     1.29     2.61    21.35     2.66    7.07   2.67  99.70
md1               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md6               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md5               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
VG_SSD-LV_BEAR002_SRV     0.00     0.00   60.00  239.00     1.30     0.99    15.68     2.80    9.35   3.32  99.40
VG_RAID0-LV_BEAR002_SRV     0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
VG_RAID10-LV_VSERVER     0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
VG_SSD-LV_RABBIT001_MNESIA     0.00     0.00    0.00  423.00     0.00     1.62     7.83     4.24    9.87   2.19  92.80

In my case, you can see sde / LV_RABBIT001_MNESIA is doing 90% utilization of disk usage. Check last number in each row to see % of whole disk utilization. As I know VG_SSD actually resides on /dev/sde I know it’s the same thing.

Note: Some iostat commands won’t show you the name of actual logical volume and will show dm-XX device instead. To see which logical volume this is, type: dmsetup ls and match last number of dm and last number of output of dmsetup command.

On specific LVM / disk

You can use utility called iotop. It will show you exact process doing most I/O operations in the server. Example of output would be:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 1475.05 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 1806 be/4 mysql         0.00 B   1440.00 K  0.00 % 19.33 % rabbitmq
  555 be/4 root          0.00 B      0.00 B  0.00 %  7.13 % [kjournald]
 1795 be/4 mysql         0.00 B      0.00 B  0.00 %  0.13 % rabbitmq
 1790 be/4 mysql         0.00 B      0.00 B  0.00 %  0.02 % rabbitmq
 1797 be/4 mysql         0.00 B      0.00 B  0.00 %  0.02 % rabbitmq
 1796 be/4 mysql         0.00 B      0.00 B  0.00 %  0.01 % rabbitmq
 1804 be/4 mysql         0.00 B      4.00 K  0.00 %  0.00 % rabbitmq
Specific files being written

To go further, we want to know names of actual files that are being written on the server by rabbitmq process. Sometimes it’s not enough to just know the process. We know that rabbitmq uses few directories where it’s installed. We can check each of them, but the one that’s making most I/O activity in my case is: /var/lib/rabbitmq. Let’s see what’s going on by executing inotifywait command which is part of inotify-tools package on CentOS.

rabbit001 ~ $ inotifywait -rme modify,move,delete,create /var/lib/rabbitmq
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/queues/2F416XM9CM1P75CLP1NL4R1OY/ MODIFY journal.jif
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/queues/4MGA9W5SBTSBHL32RA2ENN4RJ/ MODIFY journal.jif
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/queues/2F416XM9CM1P75CLP1NL4R1OY/ MODIFY journal.jif
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
/var/lib/rabbitmq/mnesia/[email protected]/msg_store_persistent/ MODIFY 496183.rdq
...

You can see rabbit is constantly modifying my queue file, causing quite a lot of I/O activity. We can sort things out from here on, since we know exactly what’s going on.

Comments

  1. By waqar

    Reply

    • Reply

      • By waqar

  2. By Anoymous

    Reply

Leave a Reply

help-hint.png
Purpose of the commenting system is to share your experience. I encourage you to post feedback with your own suggestions, ideas or optimizations regarding the topic of a blog post. What commenting system isn't for, is asking questions about similar issues of yours and requesting support for it. Blog post is provided as is and I am not here to solve all your problems. Please bear that in mind and try to avoid posting such comments. I do take privilege to remove comment from my blog for any reason whatsoever. Usually I do it when I sense a comment was posted only for spam/seo reasons or is out of blog post's topic. Thank you for reading this, now you may continue :)
 

Your email address will not be published. Required fields are marked *