imported>ThorstenStaerk |
imported>ThorstenStaerk |
| Line 1: |
Line 1: |
| − | Sometimes, for example when [[troubleshooting]] performance problems you need to understand how your system interacts with its [[hard disk]]s. You may have used [[ioMeter]] to measure your disks' speed and [[ioStat]] and [[sar]] to gather statistics. However sometimes you need to drill down deeper into the analysis, you need to understand what happens on the block level. I call this I/O sniffing. The goal is to observe packets like this:
| + | #REDIRECT [[Mindmapping]] |
| − | | |
| − | initiator XYZ requests block 4711 from device 0815
| |
| − | initiator BLA writes block 1234 to device 9876
| |
| − | block 4711 arrives from device 0815
| |
| − | | |
| − | It pays to have a good understanding about [[Bandwidth, latency and throughput]], [[block sizes]], caches, buffers, IOPS, async I/O vs sync I/O and buffered I/O vs direct I/O.
| |
| − | | |
| − | You can do I/O sniffing using the command blktrace. blktrace will show you every request that goes to the disk.
| |
| − | | |
| − | Example:
| |
| − | | |
| − | # blktrace -d /dev/sdg -o - | blkparse -i -
| |
| − | [...]
| |
| − | <abbr title="device (major/minor number)">8,96</abbr> <abbr title="processor">7</abbr> <abbr title="sequence number">106</abbr> <abbr title="timestamp">0.373952974</abbr> <abbr title="PID">11364</abbr> <abbr title="Action/Event">D</abbr> <abbr title="RWBS (D) field">W</abbr> <abbr title="starting block + blocks">0 + 8</abbr> <abbr title="process">[kworker/7:2]</abbr>
| |
| − | 8,96 7 107 0.374456639 47 C W 0 + 8 [0]
| |
| − | | |
| − | | |
| − | The RWBS(D) field can be a combination of
| |
| − | R : Read
| |
| − | W : Write
| |
| − | D : Block discard
| |
| − | B : Barrier operation
| |
| − | S : Synchronous operations
| |
| − | | |
| − | = Exploring blktrace =
| |
| − | Using the command
| |
| − | blktrace -d /dev/sdg -o - | blkparse -i - |grep -v kworker
| |
| − | shows me all operations but the kernel ones.
| |
| − | | |
| − | Now the command
| |
| − | dd if=/dev/sdg count=1
| |
| − | results in a lot of operations when I do it for the first time, but only "N" operations when I do it subsequently: The block is already in the files system cache. So let's omit the file system cache:
| |
| − | dd if=/dev/sdg count=1 iflag=direct
| |
| − | Now this results in a read operation every time I do it. iflag=direct circumvents the file system cache. With this flag it is no longer possible to set a wrong block size:
| |
| − | # dd if=/dev/sdg count=1 iflag=direct bs=8193
| |
| − | dd: error reading ‘/dev/sdg’: Invalid argument
| |
| − | But it is possible to do IO in chunks of 16 blocks:
| |
| − | dd if=/dev/sdg count=1 iflag=direct bs=8192
| |
| − | results in
| |
| − | 8,96 6 24 218.106376166 6904 D R 0 + 16 [dd]
| |
| − | | |
| − | = See also =
| |
| − | * [http://linux.die.net/man/8/blktrace blktrace man page]
| |
| − | * [http://linux.die.net/man/1/blkparse blkparse man page]
| |
| − | * http://unix.stackexchange.com/questions/105610/how-does-blktrace-work
| |
| − | * http://unix.stackexchange.com/questions/105612/why-does-blktrace-only-write-blocks-of-8
| |
| − | | |
| − | [[Category:geeky]][[Category:analysis]]
| |
