Difference between revisions of "Blocks, block devices and block sizes"

From Linuxintro
imported>ThorstenStaerk
 
(27 intermediate revisions by 4 users not shown)
Line 1: Line 1:
A [[block device]] is a [[device]] you can read blocks from. For example [[hard disks]], [[cdrom]] drives and [[floppies]] are block devices, but not the [[keyboard]]. You can receive data from the keyboard and regard them as blocks, but you cannot seek on the keyboard. You can tell a hard disk "give me block 5433", then block 7707, then block 1807 and you cannot do this with a keyboard, so, a keyboard is no block device.
+
A [[block device]] is a [[device]] you can read blocks from. For example [[hard disks]], cdrom drives and floppies are block devices, but not the [[keyboard]]. You can receive data from the keyboard and regard them as blocks, but you cannot seek on the keyboard. You can tell a hard disk "give me block 5433", then block 7707, then block 1807 and you cannot do this with a keyboard, so, a keyboard is no block device.
  
 
= Block sizes =
 
= Block sizes =
Line 9: Line 9:
 
  1+0 records in
 
  1+0 records in
 
  1+0 records out
 
  1+0 records out
  '''512 bytes (512 B)''' copied, 1.8977e-05 s, 27.0 MB/s
+
  512 bytes (512 B) copied, 1.8977e-05 s, 27.0 MB/s
 +
 
  
 
== File system block sizes ==
 
== File system block sizes ==
On the other hand, every [[file system]] needs to split up a [[partition]] into blocks to store [[file]]s and file parts. This is why there is a different block size for a file system as you can see here:
+
On the other hand, every file system needs to split up a partition into blocks to store [[file]]s and file parts. This is why there is a different block size for a file system as you can see here:
  tweedleburg:/mnt/sdb2 # stat -f .
+
  $ stat -f .
 
   File: "."
 
   File: "."
     ID: 236d62321492c2ce Namelen: 255    Type: ext2/ext3
+
     ID: 84bc409db75bdd32 Namelen: 255    Type: ext2/ext3
  Block size: '''4096'''      Fundamental block size: 4096
+
  Block size: '''''4096'''''      Fundamental block size: 4096
  Blocks: Total: 76899893  Free: 8380419   Available: 4474114
+
  Blocks: Total: 2531650    Free: 1704179   Available: 1570650
  Inodes: Total: 39075840  Free: 38013010
+
  Inodes: Total: 647168    Free: 567361
So, if you store a file in this file system, it will be stored in a 4096-byte-block, that means, even if your file only contains 5 bytes, it will take away 4096 bytes from your disk's capacity:
+
 
  tweedleburg:/mnt/sdb2 # df .
+
So, if you store a file in this file system, it will be stored in a 4096-byte-block, that means, even if your file only contains 6 bytes, it will take away 4096 bytes from your disk's capacity:
  Filesystem           '''1K-blocks'''      Used Available Use% Mounted on
+
 
  /dev/sdb2            307599572 '''274077896'''  17896456  94% /mnt/sdb2
+
  $ df .
  tweedleburg:/mnt/sdb2 # echo hallo>welt
+
  Filesystem     1K-blocks   Used Available Use% Mounted on
  tweedleburg:/mnt/sdb2 # df .
+
  /dev/sda1      10126600 '''''3309884'''''   6282600 35% /
  Filesystem           '''1K-blocks'''      Used Available Use% Mounted on
+
  $ echo hello>world
  /dev/sdb2            307599572 '''274077900'''  17896452  94% /mnt/sdb2
+
  $ ls -l world
  tweedleburg:/mnt/sdb2 # du -csh welt
+
-rw-r--r-- 1 thorsten_staerk_de thorsten_staerk_de '''''6''''' Apr  1 06:17 world
  '''4.0K'''   welt
+
$ df .
 +
  Filesystem     1K-blocks   Used Available Use% Mounted on
 +
  /dev/sda1      10126600 '''''3309888'''''  6282596 35% /
 +
  $ du -sch world
 +
  4.0K    world
 
  4.0K    total
 
  4.0K    total
  
 
== Kernel block size ==
 
== Kernel block size ==
Also the kernel has its own block size. This is relevant e.g. for vmstat. In the [http://man-wiki.net/index.php/8:vmstat vmstat man page] you find the statement
+
Also the kernel has its own block size. This is relevant e.g. for vmstat. In the [http://unixhelp.ed.ac.uk/CGI/man-cgi?vmstat vmstat man page] you find the statement
 
  All linux blocks are currently 1024 bytes.
 
  All linux blocks are currently 1024 bytes.
 +
 
So, again another block size when you work with vmstat. This is the block size the Linux kernel uses internally for caching and buffering. It is the most prominent of all block sizes.
 
So, again another block size when you work with vmstat. This is the block size the Linux kernel uses internally for caching and buffering. It is the most prominent of all block sizes.
 +
 +
== Applications ==
 +
 +
=== vmstat ===
 +
vmstat uses 1k blocks
 +
 +
=== iostat ===
 +
iostat uses 512B blocks
 +
 +
= I/O =
 +
There is
 +
* buffered I/O and direct I/O
 +
* async I/O and sync I/O
 +
 +
The difference between buffered I/O and direct I/O is that direct I/O does not use the operating system cache. During the [http://linux.die.net/man/2/open open syscall], you can say that you want direct I/O. [http://unixfoo.blogspot.com/2008/01/what-is-direct-io.html More info...]
 +
 +
The difference between async I/O and sync I/O is that async I/O reads/writes from/to a file without waiting for the data to actually arrive, but rather having the data sent to a network socket. You must use libaio for this. [http://blog.lighttpd.net/articles/2006/11/09/async-io-on-linux More info...]
 +
 +
= Benchmarking I/O =
 +
See https://www.admin-magazine.com/Archive/2016/32/Fundamentals-of-I-O-benchmarking
  
 
= How to... =
 
= How to... =
Line 40: Line 66:
 
== Find out the size of a block device ==
 
== Find out the size of a block device ==
 
To find out the size of the block device ''/dev/sda'':
 
To find out the size of the block device ''/dev/sda'':
  fdisk -l ''/dev/sda''
+
  fdisk -l /dev/sda
 +
 
 
or, a bit harder to read:
 
or, a bit harder to read:
 
  hwinfo --block
 
  hwinfo --block
Line 46: Line 73:
 
== Find out the file system stored on a block device ==
 
== Find out the file system stored on a block device ==
 
A file system will typically be stored in a partition, not directly in a block device. To find all partitions on ''/dev/sda'' use
 
A file system will typically be stored in a partition, not directly in a block device. To find all partitions on ''/dev/sda'' use
  fdisk -l ''/dev/sda''
+
  fdisk -l /dev/sda
 +
 
 
To find out what file system is stored on ''/dev/sda1'' use
 
To find out what file system is stored on ''/dev/sda1'' use
  file -s ''/dev/sda1''
+
  file -s /dev/sda1
 +
 
 +
== Find out the elevator algorithm ==
 +
The elevator algorithm can be found like this:
 +
cat /sys/devices/platform/host7/session1/target7\:0\:0/7\:0\:0\:0/block/sdi/queue/scheduler
 +
noop anticipatory [deadline] cfq
 +
 
 +
In this case, the deadline elevator algorithm is active.
 +
 
 +
Or it may be found like this:
 +
cat /sys/class/block/sdb/queue/scheduler
 +
noop deadline [cfq]
 +
 
 +
You can switch the scheduler like this:
 +
# cat /sys/class/block/sdb/queue/scheduler
 +
noop deadline [cfq]
 +
# echo "noop">/sys/class/block/sdb/queue/scheduler
 +
# cat /sys/class/block/sdb/queue/scheduler
 +
[noop] deadline cfq
  
 
= Associated commands =
 
= Associated commands =
 
* [[dd]]
 
* [[dd]]
 
* [[vmstat]]
 
* [[vmstat]]
* [http://man-wiki.net/index.php/1:iostat iostat]
+
* [http://linux.die.net/man/1/iostat iostat]
 
* [[stat]]
 
* [[stat]]
* [[fdisk]]
+
* [https://www.howtogeek.com/106873/how-to-use-fdisk-to-manage-partitions-on-linux/ fdisk]
 
* [[hwinfo]] --block
 
* [[hwinfo]] --block
 
* [[hwinfo]] --partition
 
* [[hwinfo]] --partition
Line 61: Line 107:
  
 
= See also =
 
= See also =
 +
* [[paging vs. swapping]]
 
* http://www.linuxforums.org/forum/misc/5654-linux-disk-block-size-help-please.html
 
* http://www.linuxforums.org/forum/misc/5654-linux-disk-block-size-help-please.html
 +
 +
[[Category:Concept]]

Latest revision as of 19:12, 23 May 2021

A block device is a device you can read blocks from. For example hard disks, cdrom drives and floppies are block devices, but not the keyboard. You can receive data from the keyboard and regard them as blocks, but you cannot seek on the keyboard. You can tell a hard disk "give me block 5433", then block 7707, then block 1807 and you cannot do this with a keyboard, so, a keyboard is no block device.

Block sizes

It is important to understand the ideas behind the block sizes.

Disk block sizes

Typically, a hard disk cannot read less than 512 bytes, if you want to read less, read 512 bytes and discard the rest. This is why dd reads one block à 512 bytes in the following example:

tweedleburg:~ # dd if=/dev/sda1 of=/dev/null count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 1.8977e-05 s, 27.0 MB/s


File system block sizes

On the other hand, every file system needs to split up a partition into blocks to store files and file parts. This is why there is a different block size for a file system as you can see here:

$ stat -f .
  File: "."
    ID: 84bc409db75bdd32 Namelen: 255     Type: ext2/ext3
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 2531650    Free: 1704179    Available: 1570650
Inodes: Total: 647168     Free: 567361

So, if you store a file in this file system, it will be stored in a 4096-byte-block, that means, even if your file only contains 6 bytes, it will take away 4096 bytes from your disk's capacity:

$ df .
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/sda1       10126600 3309884   6282600  35% /
$ echo hello>world
$ ls -l world 
-rw-r--r-- 1 thorsten_staerk_de thorsten_staerk_de 6 Apr  1 06:17 world
$ df .
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/sda1       10126600 3309888   6282596  35% /
$ du -sch world
4.0K    world
4.0K    total

Kernel block size

Also the kernel has its own block size. This is relevant e.g. for vmstat. In the vmstat man page you find the statement

All linux blocks are currently 1024 bytes.

So, again another block size when you work with vmstat. This is the block size the Linux kernel uses internally for caching and buffering. It is the most prominent of all block sizes.

Applications

vmstat

vmstat uses 1k blocks

iostat

iostat uses 512B blocks

I/O

There is

  • buffered I/O and direct I/O
  • async I/O and sync I/O

The difference between buffered I/O and direct I/O is that direct I/O does not use the operating system cache. During the open syscall, you can say that you want direct I/O. More info...

The difference between async I/O and sync I/O is that async I/O reads/writes from/to a file without waiting for the data to actually arrive, but rather having the data sent to a network socket. You must use libaio for this. More info...

Benchmarking I/O

See https://www.admin-magazine.com/Archive/2016/32/Fundamentals-of-I-O-benchmarking

How to...

Find out the size of a block device

To find out the size of the block device /dev/sda:

fdisk -l /dev/sda

or, a bit harder to read:

hwinfo --block

Find out the file system stored on a block device

A file system will typically be stored in a partition, not directly in a block device. To find all partitions on /dev/sda use

fdisk -l /dev/sda

To find out what file system is stored on /dev/sda1 use

file -s /dev/sda1

Find out the elevator algorithm

The elevator algorithm can be found like this:

cat /sys/devices/platform/host7/session1/target7\:0\:0/7\:0\:0\:0/block/sdi/queue/scheduler
noop anticipatory [deadline] cfq

In this case, the deadline elevator algorithm is active.

Or it may be found like this:

cat /sys/class/block/sdb/queue/scheduler 
noop deadline [cfq]

You can switch the scheduler like this:

# cat /sys/class/block/sdb/queue/scheduler 
noop deadline [cfq] 
# echo "noop">/sys/class/block/sdb/queue/scheduler
# cat /sys/class/block/sdb/queue/scheduler 
[noop] deadline cfq

Associated commands

See also