Difference between revisions of "Dd"

From LinuxIntro
Jump to: navigation, search
m (cat)
 
(3 intermediate revisions by one user not shown)
Line 15: Line 15:
 
== Create a backup ==
 
== Create a backup ==
 
Say we have a harddisk /dev/sda that we want to backup entirely (sector-by-sector) to a USB volume /dev/sdb1, mounted on /mnt/sdb1. We call this a ''dump'' or an ''image'' of /dev/sda. The dump shall be named ''backup.img''. Here is the dd command:
 
Say we have a harddisk /dev/sda that we want to backup entirely (sector-by-sector) to a USB volume /dev/sdb1, mounted on /mnt/sdb1. We call this a ''dump'' or an ''image'' of /dev/sda. The dump shall be named ''backup.img''. Here is the dd command:
 +
<source>
 
  dd if=/dev/sda of=/mnt/sdb1/backup.img
 
  dd if=/dev/sda of=/mnt/sdb1/backup.img
 +
</source>
 
In this command, '''if''' stands for '''i'''nput '''f'''ile and '''of''' for '''o'''utput '''f'''ile.
 
In this command, '''if''' stands for '''i'''nput '''f'''ile and '''of''' for '''o'''utput '''f'''ile.
  
Line 21: Line 23:
 
To restore this backup, we boot from a live CD and do the command vice versa.  
 
To restore this backup, we boot from a live CD and do the command vice versa.  
 
'''This can overwrite all content on your harddisk, this is the intention.'''
 
'''This can overwrite all content on your harddisk, this is the intention.'''
 +
<source>
 
  dd if=/mnt/sdb1/backup.img of=/dev/sda
 
  dd if=/mnt/sdb1/backup.img of=/dev/sda
 +
</source>
  
 
== Clone a harddisk ==
 
== Clone a harddisk ==
 
To clone a disk A to B, both disks need to have the same capacity. It is very convenient for USB disks. Say our USB disk source is called /dev/sdb and the target is called /dev/sdc. Do it like this:
 
To clone a disk A to B, both disks need to have the same capacity. It is very convenient for USB disks. Say our USB disk source is called /dev/sdb and the target is called /dev/sdc. Do it like this:
 +
<source>
 
  dd if=/dev/sdb of=/dev/sdc
 
  dd if=/dev/sdb of=/dev/sdc
 +
</source>
 
Now if sdc has a bigger capacity, this capacity will be lost because the file system is not aware of it.
 
Now if sdc has a bigger capacity, this capacity will be lost because the file system is not aware of it.
  
 
== Transfer a disk image ==
 
== Transfer a disk image ==
 
To transfer a disk image over the network to a computer named ''target'', use
 
To transfer a disk image over the network to a computer named ''target'', use
  dd if=/dev/sdb | ssh root@''target'' "(cat >backup.img)"
+
<source>
 +
  dd if=/dev/sdb | ssh root@target "(cat >backup.img)"
 +
</source>
  
 
= create an iso image of a CD =
 
= create an iso image of a CD =
 
To create an iso image of a CD, read it block-by-block and save the blocks to a file. In case of an error, dd is not supposed to abort:
 
To create an iso image of a CD, read it block-by-block and save the blocks to a file. In case of an error, dd is not supposed to abort:
 +
<source>
 
  dd if=/dev/cdrom of=cdimage.iso conv=noerror
 
  dd if=/dev/cdrom of=cdimage.iso conv=noerror
 +
</source>
  
 
= create a disk image =
 
= create a disk image =
 
To create an empty image of a harddisk, e.g. for [[qemu]] [[virtualization]]:
 
To create an empty image of a harddisk, e.g. for [[qemu]] [[virtualization]]:
 +
<source>
 
  dd if=/dev/zero of=harddisk.img bs=516096 seek=6241 count=0
 
  dd if=/dev/zero of=harddisk.img bs=516096 seek=6241 count=0
 +
</source>
 
creates a 3GB-Image. For a 6GB-Image, use seek=12482.
 
creates a 3GB-Image. For a 6GB-Image, use seek=12482.
  
 
= rescue a file that contains bad blocks =
 
= rescue a file that contains bad blocks =
 
If your favorite movie or song cannot be played any longer because the file is corrupt, you can use dd to ignore the corrupt part:
 
If your favorite movie or song cannot be played any longer because the file is corrupt, you can use dd to ignore the corrupt part:
 +
<source>
 
  dd if=movie.avi of=rescued_movie.avi conv=noerror
 
  dd if=movie.avi of=rescued_movie.avi conv=noerror
 +
</source>
  
 
= analyze your disk =
 
= analyze your disk =
DD is great to learn about your system. To analyze your disk by displaying selected [[blocks]], in this case block 1001 use:
+
With dd you can read any byte from your disk. For example to read the first 40 bytes from your first disk, issue
 +
<source>
 +
dd if=/dev/sda bs=40 count=1
 +
</source>
 +
The output will be displayed on the console. It will look about like this:
 +
<source>
 +
tweedleburg:~ # dd if=/dev/sda bs=40 count=1
 +
3���؎м|��W���������RR�A��U1�1+0 records in
 +
1+0 records out
 +
40 bytes (40 B) copied, 4.6373e-05 s, 863 kB/s
 +
</source>
 +
You see there are many glyphs that are no characters or numbers. To make sense out of this, [[pipe]] it to [[hexdump]]:
 +
<source>
 +
tweedleburg:~ # dd if=/dev/sda bs=40 count=1 | hexdump -C
 +
1+0 records in
 +
1+0 records out
 +
40 bytes (40 B) copied, 5.4665e-05 s, 732 kB/s
 +
00000000  33 c0 fa 8e d8 8e d0 bc  00 7c 89 e6 06 57 8e c0  |3........|...W..|
 +
00000010  fb fc bf 00 06 b9 00 01  f3 a5 ea 1f 06 00 00 52  |...............R|
 +
00000020  52 b4 41 bb aa 55 31 c9                          |R.A..U1.|
 +
00000028
 +
</source>
 +
Looks much better, right? Now you know the first byte on your first harddisk is 33 (hexadecimal).
 +
 
 +
To analyze your disk by displaying selected [[blocks]], in this case block 1001 use:
 +
<source>
 
  dd if=/dev/sdc1 count=1 skip=1000
 
  dd if=/dev/sdc1 count=1 skip=1000
 +
</source>
  
 
= Create your own bootloader =
 
= Create your own bootloader =
 
To create your own operating system by dumping your bootloader to the boot sector of a bootable disk image use
 
To create your own operating system by dumping your bootloader to the boot sector of a bootable disk image use
 +
<source>
 
  dd conv=notrunc if=bootloader of=[[qemu]].img
 
  dd conv=notrunc if=bootloader of=[[qemu]].img
 +
</source>
  
 
= benchmark the throughput of your disks =
 
= benchmark the throughput of your disks =
 
To benchmark the throughput of your disk ''/dev/sda1'', e.g. for different [[block size]]s, proceed like this:
 
To benchmark the throughput of your disk ''/dev/sda1'', e.g. for different [[block size]]s, proceed like this:
 +
<source>
 
  # dd if=/dev/sda1 of=/dev/null bs=512 count=1000000
 
  # dd if=/dev/sda1 of=/dev/null bs=512 count=1000000
 
  1000000+0 records in
 
  1000000+0 records in
Line 63: Line 106:
 
  1000000+0 records out
 
  1000000+0 records out
 
  4096000000 bytes (4.1 GB) copied, 36.0667 s, 114 MB/s
 
  4096000000 bytes (4.1 GB) copied, 36.0667 s, 114 MB/s
 +
</source>
 
However, make sure you have read [[Background:How caching works]] first otherwise you will be surprised by a mysterious accelleration like this:
 
However, make sure you have read [[Background:How caching works]] first otherwise you will be surprised by a mysterious accelleration like this:
 +
<source>
 
  # dd if=/dev/sda1 of=/dev/null bs=512 count=1000000
 
  # dd if=/dev/sda1 of=/dev/null bs=512 count=1000000
 
  1000000+0 records in
 
  1000000+0 records in
Line 72: Line 117:
 
  1000000+0 records out
 
  1000000+0 records out
 
  512000000 bytes (512 MB) copied, 1.09851 s, 466 MB/s
 
  512000000 bytes (512 MB) copied, 1.09851 s, 466 MB/s
 +
</source>
  
 
= get the progress =
 
= get the progress =
 
During a long dd run you may want to know the progress. dd will output it for you if it receives a special signal, USR1. So,  
 
During a long dd run you may want to know the progress. dd will output it for you if it receives a special signal, USR1. So,  
 
* find out the number of SIGUSR
 
* find out the number of SIGUSR
 +
<source>
 
   # kill -l
 
   # kill -l
 
   1) SIGHUP      2) SIGINT      3) SIGQUIT      4) SIGILL      5) SIGTRAP
 
   1) SIGHUP      2) SIGINT      3) SIGQUIT      4) SIGILL      5) SIGTRAP
Line 89: Line 136:
 
  53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
 
  53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
 
  58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
 
  58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
 +
</source>
 
* send SIGUSR to all dd [[process]]es:
 
* send SIGUSR to all dd [[process]]es:
 +
<source>
 
  killall -10 [[dd]]
 
  killall -10 [[dd]]
 +
</source>
 
The output will read like this:
 
The output will read like this:
 +
<source>
 
  2989696+0 records in
 
  2989696+0 records in
 
  2989696+0 records out
 
  2989696+0 records out
 
  1530724352 bytes (1.5 GB) copied, 583.336 s, 2.6 MB/s
 
  1530724352 bytes (1.5 GB) copied, 583.336 s, 2.6 MB/s
 +
</source>
  
 
[[Category:low-level]]
 
[[Category:low-level]]
 
[[Category:command]]
 
[[Category:command]]

Latest revision as of 09:03, 13 April 2020

dd is a utility to create a disk dump by reading every single block on a disk, e.g. your hard drive. However, its architecture is laid out so it can do much more than creating a dump. Here is what dd can do for you:

  • manage a disk backup
    • create a backup from a disk to a file
    • restore a backup from a file to a disk
    • clone a harddisk
    • create a disk image and transfer it over the network
  • create an iso image of a CD
  • rescue a file that contains bad blocks
  • analyze your disk by displaying selected blocks
  • create your own operating system by dumping your bootloader to the boot sector
  • benchmark the throughput of your disks

Contents

Disk Backup

Create a backup

Say we have a harddisk /dev/sda that we want to backup entirely (sector-by-sector) to a USB volume /dev/sdb1, mounted on /mnt/sdb1. We call this a dump or an image of /dev/sda. The dump shall be named backup.img. Here is the dd command:

 dd if=/dev/sda of=/mnt/sdb1/backup.img

In this command, if stands for input file and of for output file.

Restore a backup

To restore this backup, we boot from a live CD and do the command vice versa. This can overwrite all content on your harddisk, this is the intention.

 dd if=/mnt/sdb1/backup.img of=/dev/sda

Clone a harddisk

To clone a disk A to B, both disks need to have the same capacity. It is very convenient for USB disks. Say our USB disk source is called /dev/sdb and the target is called /dev/sdc. Do it like this:

 dd if=/dev/sdb of=/dev/sdc

Now if sdc has a bigger capacity, this capacity will be lost because the file system is not aware of it.

Transfer a disk image

To transfer a disk image over the network to a computer named target, use

 dd if=/dev/sdb | ssh root@target "(cat >backup.img)"

create an iso image of a CD

To create an iso image of a CD, read it block-by-block and save the blocks to a file. In case of an error, dd is not supposed to abort:

 dd if=/dev/cdrom of=cdimage.iso conv=noerror

create a disk image

To create an empty image of a harddisk, e.g. for qemu virtualization:

 dd if=/dev/zero of=harddisk.img bs=516096 seek=6241 count=0

creates a 3GB-Image. For a 6GB-Image, use seek=12482.

rescue a file that contains bad blocks

If your favorite movie or song cannot be played any longer because the file is corrupt, you can use dd to ignore the corrupt part:

 dd if=movie.avi of=rescued_movie.avi conv=noerror

analyze your disk

With dd you can read any byte from your disk. For example to read the first 40 bytes from your first disk, issue

 dd if=/dev/sda bs=40 count=1

The output will be displayed on the console. It will look about like this:

 tweedleburg:~ # dd if=/dev/sda bs=40 count=1
 3���؎м|��W���������RR�A��U1�1+0 records in
 1+0 records out
 40 bytes (40 B) copied, 4.6373e-05 s, 863 kB/s

You see there are many glyphs that are no characters or numbers. To make sense out of this, pipe it to hexdump:

 tweedleburg:~ # dd if=/dev/sda bs=40 count=1 | hexdump -C
 1+0 records in
 1+0 records out
 40 bytes (40 B) copied, 5.4665e-05 s, 732 kB/s
 00000000  33 c0 fa 8e d8 8e d0 bc  00 7c 89 e6 06 57 8e c0  |3........|...W..|
 00000010  fb fc bf 00 06 b9 00 01  f3 a5 ea 1f 06 00 00 52  |...............R|
 00000020  52 b4 41 bb aa 55 31 c9                           |R.A..U1.|
 00000028

Looks much better, right? Now you know the first byte on your first harddisk is 33 (hexadecimal).

To analyze your disk by displaying selected blocks, in this case block 1001 use:

 dd if=/dev/sdc1 count=1 skip=1000

Create your own bootloader

To create your own operating system by dumping your bootloader to the boot sector of a bootable disk image use

 dd conv=notrunc if=bootloader of=[[qemu]].img

benchmark the throughput of your disks

To benchmark the throughput of your disk /dev/sda1, e.g. for different block sizes, proceed like this:

 # dd if=/dev/sda1 of=/dev/null bs=512 count=1000000
 1000000+0 records in
 1000000+0 records out
 512000000 bytes (512 MB) copied, 5.16588 s, 99.1 MB/s
 # dd if=/dev/sda1 of=/dev/null bs=4096 count=1000000
 1000000+0 records in
 1000000+0 records out
 4096000000 bytes (4.1 GB) copied, 36.0667 s, 114 MB/s

However, make sure you have read Background:How caching works first otherwise you will be surprised by a mysterious accelleration like this:

 # dd if=/dev/sda1 of=/dev/null bs=512 count=1000000
 1000000+0 records in
 1000000+0 records out
 512000000 bytes (512 MB) copied, 5.32254 s, 96.2 MB/s
 # dd if=/dev/sda1 of=/dev/null bs=512 count=1000000
 1000000+0 records in
 1000000+0 records out
 512000000 bytes (512 MB) copied, 1.09851 s, 466 MB/s

get the progress

During a long dd run you may want to know the progress. dd will output it for you if it receives a special signal, USR1. So,

  • find out the number of SIGUSR
  # kill -l
  1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
  6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     '''10''') SIGUSR1
 11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
 16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
 21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
 26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
 31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
 38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
 43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
 53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
 58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
 killall -10 [[dd]]

The output will read like this:

 2989696+0 records in
 2989696+0 records out
 1530724352 bytes (1.5 GB) copied, 583.336 s, 2.6 MB/s