Virtual block device

If a user has experience installing Linux on a virtual machine, they may find it difficult to distinguish between what is virtual and what is real. The entire operating system can be virtualized, including virtual block devices such as CD-ROMs, hard disks, and filesystems. For Linux, this virtualization is a common and essential feature, and Linux also comes with various built-in tools for virtualization and real device interaction.

The philosophy of UNIX/Linux is "Everything is a file," and the key to this concept is that a "file" can also represent a physical storage device like "/dev/sda." Therefore, in Linux, there is only a thin line between files and physical devices, and understanding the interchangeability of virtual and real can lead to infinite possibilities.

The Heart Sutra states, "Form is not different from emptiness, emptiness is not different from form." In this context, "form" refers to physical objects, and "emptiness" refers to the virtual. In simpler terms, it means "virtual and real are the same." In the world of UNIX/Linux, physical storage devices and the "device files" representing them embody the principle of "form is emptiness, emptiness is form."

dd : Physical to virtual conversion
In Windows, to convert a CD/DVD to an ISO file, commonly used tools are "WinISO" or "UltraISO." For complete disk duplication, commercial software like "Norton GHOST" or "Acronis True Image" is often used. However, in Linux, there is a powerful and widely known tool called "dd," which provides similar functionality to most of those tools and is free to use. The only difference is that dd does not have a graphical interface and requires command-line usage. Interestingly, the name dd has multiple interpretations despite its powerful features.

The primary purpose o fdd" is to read data and perform conversions, somewhat similar to tr, as shown in the example below:

The parameter "conv=ucase" converts lowercase to uppercase, and the opposite parameter "conv=lcase" converts uppercase to lowercase.

dd can also be used with input and output file options, represented by "if=" (input file) and "of=" (output file), respectively.

Though dd may seem simple at first glance, it is capable of more than just character conversions. It can be used to convert physical devices to virtual ones, bridging the gap between real and virtual. dd is able to convert physical devices to virtual device files, often referred to as "image files" or "disk images," which can be treated as real media when needed.

For instance, the following example converts a physical CD-ROM (CentOS 7 installation disc) to a virtual image-file (ISO file):

In this example, the parameter "bs" indicates the block size for reading/writing, and the "status=progress" parameter is added to show the conversion progress.

Similarly, dd can also be used to convert physical partitions or floppy disks into image files.

Once an image-file is created, it can be mounted as a virtual device or restored using losetup, the reverse of dd.

IT professionals who have used Norton Ghost to copy/backup Windows systems can use dd as a free alternative.

Sometimes, the MBR (Master Boot Record) or GPT (GUID Partition Table) of a hard disk may become corrupted due to malware or unknown reasons, preventing the system from booting. In such cases, dd can be used to create a recovery image-file, backing up the MBR or GPT table, which can later be used for disaster recovery.

MBR is stored in the first 512 bytes of the hard disk, and GPT can be backed up by saving the first 34 Logical Block Addresses (LBA). The following examples demonstrate backing up and restoring the MBR and GPT:

In the above example," bs=512" is used because the MBR table is only 512 bytes in size. The "count" parameter specifies how many times to execute the "bs" block size.

dd can also be used to completely erase hard drive data. This is because mkfs only rebuilds the inodes when formatting an ext2/ext3/ext4 partition, and does not erase the blocks. This means that someone with the right tools may be able to recover the data. To completely erase a hard drive, you can use dd if=/dev/zero of=/dev/sda to fill the entire hard drive with zeros.

When necessary, the virtual partition can be added to "/etc/fstab" for automatic mounting on startup. Such virtual image-files can be used not only as partitions but also as disks for partitioning using tools like parted.

Using dd provides the added benefit of time measurement, making it a useful tool for testing the speed of a hard disk. For example, the following commands can be used to test

^ back on top ^

When mounting an image file using mount -v -o loop, you might encounter the term "loop device." So, what is a "loop device"? A "loop device" is a virtual block device. Loop devices are available by default, usually eight of them, numbered from "/dev/loop0" to "/dev/loop7". The number of loop devices supported may vary among different Linux distributions. These devices cannot be directly used until you specify a file (an image file) to associate with the loop device. The tool used to associate a file with a loop device is called losetup (loop device setup).

The usage of losetup is as follows: losetup [-option] [/dev/loop#] device (where # is a number). For instance, losetup /dev/loop0 200M-image associates the virtual block device file "/dev/loop0" with the image file 200M-image. At this point, "/dev/loop0" becomes an interface for the image file. The operating system treats this virtual block device file, such as /dev/loop#, just like a real storage device, making operations indistinguishable from those performed on physical storage devices.

When using the mount -o loop command, it is actually utilizing the losetup command behind the scenes. If you were to break down the process without using mount -o loop, it would look something like this:

To detach the association between losetup and an image file, you can use the "-d" option. For instance, losetup -d /dev/loop0 would disconnect the association between "/dev/loop0" and the image file.

The "-v" option displays whether a loop device is associated with an image file or not. For example, using losetup -v /dev/loop3 or losetup -a scans from" /dev/loop0" to "/dev/loop7" to show the current associations. Another option, "-f", finds the next available unused loop device automatically, eliminating the need to specify a loop device explicitly.

In the section about dd, there was a demonstration of creating an empty image file and then immediately formatting it, resulting in a "not a block special device" warning. This is because an empty image file created with dd pretends to be a storage device but is quickly recognized as a sham. If you use losetup to associate this empty image file, it will closely resemble a real block device, even appearing in the output of blkid. Here's an example:

After using losetup on "/dev/loop0", it behaves just like a real hard disk. When you use mkfs /dev/loop0 to format, there will be no "not a block special device" warning as before when formatting the image file directly. You can also use fdisk /dev/loop0 to partition it (for image files larger than 2TB, you might need to use parted).

From these experiments, it's evident that losetup transforms virtual image files into actual devices. By combining dd and losetup, you can seemingly create an additional eight hard disks (from "/dev/loop0" to "/dev/loop7") without dealing with the hassle of handling physical disks. You can use these virtual devices to experiment with concepts like LVM and RAID without risking your system stability.

If you've used dd to backup a hard disk with multiple partitions, and you want to separate individual partitions, you can use the offset option "-o". Here's an example of a split image file with three partitions:

When an image file contains more than one partition, and the "-o" (offset) option is not used, it can lead to incorrect usage and potential damage to the image file when accessing it.

Since the offset unit for the "-o" option is in bytes, you can use parted to automatically convert the offset units to bytes for use with the losetup -o option. Here's an example of how to do this:

In the example above, the starting bytes for partitions #1 to #3 are 16384, 20971520, and 52428800 respectively. These are the values to be used as the offset for the loop devices. You can set up the loop devices to correspond to these partitions as shown below:

Because mount -o loop essentially executes losetup in the background, additional loop devices may be used (you can scan the loop devices with losetup -a). After applying the correct offset handling, "/dev/loop0" to "/dev/loop2" will virtually represent the three partitions of the image file.

The offset values for partitions can also be calculated using sector values provided by fdisk, as shown in the following example:

The output from fdisk -lu is in sectors (with one sector being 512 bytes for hard disks, but possibly 2048 or 512 bytes for optical discs). The "Units" information in the fdisk -lu output specifies this sector size. In the example above, partition #1 starts at sector 32, which translates to a starting byte of 32 x 512 = 16384. This result should align with the output from parted DEVICE unit B print.

• kpartx : Mapping partitions in disk images to loop devices
  When using dd to back up a hard disk with multiple partitions and using losetup -o to separate the partitions, calculating the offset values and handling unformatted partitions can be quite cumbersome. An alternative tool called kpartx can simplify this process.

  The usage of kpartx is similar to losetup, but you don't need to worry about specifying offset values or loop devices. The usage is: kpartx [-option] image-file. The "-a" option creates links between the image file and loop devices, the "-d" option removes these links, and the "-v" option displays detailed information about loop device mappings.

  In the following example, we have an image file with four partitions, and we use kpartx to automatically map the image file to loop devices:

  Example:

  
  

  # kpartx -av disk-image add map loop0p1 : 0 30688 linear /dev/loop0 32 ←Mapping partition 1 to loop0p1 add map loop0p2 : 0 51200 linear /dev/loop0 30720 ←Mapping partition 2 to loop0p2 add map loop0p5 : 0 81888 linear /dev/loop0 81952 ←Mapping partition 5 to loop0p5 add map loop0p6 : 0 40928 linear /dev/loop0 163872 ←Mapping partition 6 to loop0p6

  
  

  After using kpartx -a image-file as shown above, if you use losetup -a, you'll notice that one loop device is occupied. However, even though the image file contains four partitions, only one loop device is used. This is because the partitions are mapped to files in the directory "/dev/mapper", where you'll find files named" loop[0-7]p[0-9]".

  These mapped files "/dev/mapper/loop[0-7]p[0-9]" can be used as if they were physical hard disks. You can format and mount them just like real hard disks. Here's an example:

  Example:

  # kpartx -av disk-image # ls /dev/mapper/loop* ←Check the mapped file names /dev/mapper/loop0p1   /dev/mapper/loop0p5 /dev/mapper/loop0p2   /dev/mapper/loop0p6 # mkfs -j /dev/mapper/loop0p1 ←format partition #1 # mount /dev/mapper/loop0p5 /media ←mount partition #5

  
  To remove the mappings created in the above example, us kpart -dv disk-image.

^ back on top ^

We've all experienced situations in Windows where running too many applications or using a resource-intensive software slows down the system, and the disk LED keeps flashing. This is often due to insufficient memory, and Windows resorts to using the hard disk as virtual memory. In Windows, this technique is known as "paging." However, traditional mechanical hard disks are significantly slower than memory, causing the system to become sluggish when using the hard disk as auxiliary memory. While it's not ideal, this approach prevents the system from crashing or becoming unresponsive.

Unix/Linux uses a similar concept called "swap." Swap involves utilizing the hard disk space as temporary memory, essentially swapping space for time. However, the question of whether to use swap, how much to allocate for swap, or when to use it remains a debated topic. Generally, if the system has ample physical memory, the need to use swap is minimized. Consequently, a smaller swap size can be set. Conversely, a larger swap size is advisable if physical memory is limited. If the system uses hibernation, where memory data is written to the disk, the swap space must be at least as large as the system's memory.

Linux industry leader Red Hat recommends a minimum swap size of 256MB or more. Here are their recommended swap sizes based on system memory, along with considerations for hibernation:

Traditionally, swap space was created using partitions. However, modern Linux systems can also use swap files, similar to Windows' paging file. Each approach has its advantages and disadvantages. Swap partitions generally offer better performance, but resizing them can be more complicated. On the other hand, swap files might have slightly lower performance (due to potential non-contiguous allocation) but are easier to resize. To check if swap is enabled on your system and its size, you can use the command swapon -s (equivalent to cat /proc/swaps).

• swap-partition
  If a partition serves as a swap partition and has corresponding settings in the "/etc/fstab" file, it will automatically be enabled during system startup.
  
  Example:

  # cat /etc/fstab | grep 'swap' LABEL=SWAP-sda3         swap                    swap    defaults        0 0

  
  
  • swapon : Enabling swap
    If you're observant and use the mount or df commands, you'll notice that a swap partition is not actually mounted. Enabling a swap partition is not done using mount, but rather a separate command called swapon. The "-a" option enables swaps according to the settings in "/etc/fstab."

    If a swap partition is not configured in "/etc/fstab," you can use swapon later to enable it. The syntax for swapon is either swapon DEVICE (where "DEVICE" is the device file) or swapon -L LABEL (where "LABEL" is the filesystem label).

    Example:

    # swapon -a ←Enable swap according to settings in "/etc/fstab" # swapon /dev/sdb3 ←Enable swap on the device "/dev/sdb3" # swapon -L SWAP-sda3 ←Enable swap with the label "SWAP-sda3"

    
    
  • swapoff : Disabling swap
    To disable swap, you can use the swapoff command. The usage and options of swapoff are similar to swapon, but it reverses the action. For instance, swapoff -a disables swaps according to the settings in "/etc/fstab."
    
    Example:

    # swapoff /dev/sdb3 ←Disable swap on the device "/dev/sdb3"

    
    
  
  If you didn't configure a swap partition during Linux installation or want to adjust its size, follow these steps:
  
  1. If you already have a swap partition and want to adjust its size, you need to delete the existing partition and recreate it. Deleting a partition can be done using the d command in fdisk or the rm command in parted (Note: Linux also supports using a single dedicated swap partition, so you don't necessarily need to delete the existing one.)
  2. Create the new swap partition. For fdisk, change the partition type to 82_(HEX) using the t command. For parted, use mkpart with fs-type "linux-swap."
  3. After creating the swap partition, format it using the mkswap command (e.g., mkswap /dev/sdb1).
  4. Update the "/etc/fstab" file with the new swap partition settings.
  5. Run the swapon -a command, or restart the system. Then use either cat /proc/swaps or swapon -s to check if the swap partition has been enabled properly.
  6. Using a dedicated hard disk for a swap partition will be faster than having multiple partitions, each with a swap partition. If speed is a priority, consider using a solid-state drive (SSD).
• swap-file : Using a file for swap
  Linux also allows you to use a file as swap space, similar to how it's done in Windows XP or Windows 7, where the file is typically named "pagefile.sys" and located in the root directory.
  
  
  • mkswap : Formatting swap
    If you didn't reserve a swap partition during the installation of the system and later find the need for swap space, you can create a swap file using dd and then format it using mkswap.

    Here's how you can create a 512 MiB swap file using dd:

    Example:

    # dd if=/dev/zero of=/mnt/512MiB_swap bs=512M count=1 ←Create a 512M swap file

    
    

    Once the swap file is generated, you're halfway there. Next, you need to set permissions on the file so that others can't read or write to it. Then, format it as swap using mkswap and finally, enable it using swapon.

    Example:

    # chmod 600 /mnt/512MiB_swap ←←Change permissions to "rw-------" # mkswap /mnt/512MiB_swap ←Format as swap # swapon /mnt/512MiB_swap ←Enable swap # cat /proc/swaps ←erify that swap is enabled Filename                            Type            Size    Used    Priority /mnt/512MiB_swap                    file            524280  0       -1

    
    

    If you want to enable swap at boot time, add the following configuration to "/etc/fstab":

    Example:

    # cat /etc/fstab | grep 'swap' /mnt/512MiB_swap         swap                    swap    defaults        0 0

  
  swappiness : Adjusting the aggressiveness of swap usage
  "Swappiness" controls how aggressively the system uses swap space. You can adjust the swappiness value to match your system's behavior. If you have a system with limited RAM, you might want it to use swap more often, while if you have ample RAM, you might prefer it to use swap less often.
  
  The swappiness value ranges from 0 to 100, where 0 means the system will try to use RAM as much as possible, and 100 means it will aggressively use swap. To check the current swappiness value, you can use:
  :
  Example:

  # cat /proc/sys/vm/swappiness ←Display the swappiness value 60

  
  The default swappiness value is typically set to 60. If you want to change it, you can edit "/etc/sysctl.con"" and add the line "vm.swappiness = n" (where n is a value between 0 and 100). After rebooting, the new swappiness value will take effect. Here's an example of setting swappiness to 30:
  
  

  # cat /etc/sysctl.conf | grep 'swappiness' vm.swappiness=30 ←his line sets swappiness to 30

^ back on top ^