User:Maffblaster/Infrastructure/Disk images
When working with small systems (such as SoCs, older slower systems, or embedded systems), it is useful to know how to manipulate disk images. This article will attempt to flesh out different creation and capture methods, mounting/attaching, conversion to run via virtual machines. This can be useful when attempting routine maintenance, package updates, etc into a disk image.
When a Linux-based operating system contained within the disk image is the same as the host computer architecture, it is simple to chroot into the disk image. In cases where the host computer architecture is different, then a qemu can be used to emulate the target system architecture and provide a static chroot.
Images
Create
Depending on the software which will used for operations, disk images can be created using a variety of tools.
Simple tooling
It is important to note when using disk image creation methods for the raw format type, the full size of the created file will be written to disk.
dd
dd, which is included as part of the @system set, will be present for all Gentoo supported computer architectures. It is possible to create a disk image of the raw format using this tool in a multitude of ways. A simple method that does not require mathematics is to use the count= and seek= parameters to specify the scope of the file to be created.
For a 20 GB file:
user $dd if=/dev/zero of=disk_image.img bs=1 count=0 seek=20Gfallocate
fallocate, which is included as part of the @system set, will be present for Gentoo supported computer architectures. Raw disk images can be quickly created using this tool:
user $fallocate --length 20GiB disk_image.imgSpecialty tooling
Specialty tooling for manipulating disk images generally associated with and provided by virtual machine software suites such as Qemu or VirtualBox.
qemu
When working with qemu-img (from app-emulation/qemu), a 20 GiB raw disk image can be created via:
user $qemu-img create disk_image.img 20GAn advantage of working with qemu-img is that it can directly create disk images in format types specialized for virtual machines. For example, to create an image in qcow2 format, which is qemu's own disk image format type, issue:
user $qemu-img create -f qcow2 disk_image.img.qcow2 20GFormatting 'disk_image.img.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=21474836480 lazy_refcounts=off refcount_bits=16
Notice that qcow2 will utilize zlib compression by default, which is a space-saving advantage when compared to raw disk images. Note that using a non-raw disk image format will require conversion back to raw format before it will be bootable from a physical storage medium (SD card, SSD, etc.).
Partition, format, mount, and chroot
To partition and format the disk image use a standard partitioning utility such as parted or fdisk. Parted can be used specify filesystems during the partitioning phase, whereas fdisk cannot.
Once partitioning has been finished, follow the mount steps below using losetup to expose the partitions contained within the disk image to the host system. The partitions may need initial formatting using the appropriate mkfs utility, then mounted using the mount command before a chroot or qemu-user chroot will be possible.
Inspection
Using the file tool (from sys-apps/file) may provide information to indicate that the target file is a disk image, but fails to provide additional details about the image.
user $file disk_image.imgdisk_image.img: DOS/MBR boot sector; partition 1 : ID=0xee, start-CHS (0x0,0,2), end-CHS (0x3ff,255,63), startsector 1, 41943039 sectors, extended partition table (last)
qemu-img
Disk image format can be determined using qemu-img:
user $qemu-img info disk_image.imgimage: disk_image.img
file format: raw
virtual size: 20 GiB (21474836480 bytes)
disk size: 4 KiB
Child node '/file':
filename: myimage.img
protocol type: file
file length: 20 GiB (21474836480 bytes)
disk size: 4 KiB
Capture
There are many tools available in Gentoo to capture disk images. Capturing an image is nothing more than making backup of the system. Generally this involves physically connecting the disk to a system and creating a file from the disk image. Starting from easier and faster methods, and moving to harder and slower methods:
- Connect the disk by inserting it into an external USB device, then connecting the USB device to a system.
- Connect the disk directly to the system's motherboard.
- Connect the disk to another system, and then send the data to a new system
See the Backup article for more information on creating backups in general.
Once the disk is connected choose a tool to use to capture the disk image. For this example, a variant of dd will be used to ensure the integrity of the backup.
It is important to verify the initial capture completes with perfect success to not encounter corruption issues.
When capturing raw disk images, the source disk will likely need to be smaller than the destination disk. For example, if the source of the backup is 500 GB, the destination will need at least 500 GB free space in order for the disk image to be captured successfully. There are some exceptions to the free space rule depending on the tool used for capture. Some tools will compress the disk image as it is captured, which can save much space in the long run, however it is likely the capture will have to be decompressed in order to be accessed in a useful manner.
ddrescue
In the following example, ddrescue is used to capture one disk as a disk image.
root #ddrescue -f -n /dev/sdb /root/sdb_rescue.img /root/sdb_rescue.logIf all goes well, the captured disk image will be available at /root/sdb_rescue.img.
Mount
After the disk image has been captured, it is typical to mount the image as loop devices.
Discovery
It is wise to first check to see what loop devices are presently in use:
root #losetup --listThe following command can be used to attach to either a single or multi-partition disk image to a loop device:
root #losetup --partscan --find sda1.imgListing the loop devices again should show something like the following:
root #losetup --listNAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop0 0 0 0 1 /home/larry/sda1.img 0 512
With the image attached to a loop device, is it now possible to run fsck commands on the image, or mount the image at a mount point.
View available partitions in the image by querying the loop device:
root #ls /dev/loop0p?/dev/loop0p1 /dev/loop0p2
Filesystem types for each partition can be checked or resized using either a disk partitioning utility such as fdisk, parted, or gparted on the loop device as a whole.
root #fdisk --list /dev/loop0Disk /dev/loop0: 100 MiB, 104857600 bytes, 204800 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xd414f95e Device Boot Start End Sectors Size Id Type /dev/loop0p1 40960 143359 102400 50M c W95 FAT32 (LBA) /dev/loop0p2 143360 204799 61440 30M 83 Linux
root #parted /dev/loop0 printModel: Loopback device (loopback) Disk /dev/loop0: 105MB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 21.0MB 73.4MB 52.4MB primary fat16 lba 2 73.4MB 105MB 31.5MB primary ext4
Opening gparted for an easy online resize:
root #gparted -a /dev/loop0GParted 1.4.0 configuration --enable-online-resize libparted 3.5
btrfs images
Presuming the sda1.img file is a btrfs filesystem, the image can be checked or repaired just like any other btrfs device. See the Troubleshooting section below on repairing dirty filesystems.
Mount the image:
root #mount -o loop -t btrfs /home/larry/sda1.img /mnt/imageUnmount
Unmounting the image is typically no different any other unmount. In a rare instance, it may be necessary to lazily unmount an image when there's heavy disk I/O.
root #umount /mnt/imageDetach
If the image was attached to a loop device, it is proper to detach when finished to free up the loop device for other uses:
root #losetup --detach /dev/loop0Convert
Converting raw disk image to VM disk image
Converting the raw image (.img) file to qcow2 format (requires app-emulation/qemu):
root #qemu-img convert -f raw -O qcow2 vm_sdd.img vm_sdd.img.qcow2Trim
Resizing a partition inside a disk image file does not decrease the allocated, on disk size of the disk image file. However, after shrinking the size of the partition(s) inside the disk image, the end of the (uncompressed) file can be trimmed off to save space. Compression can be used to save an additional amount of space. This can be performed by determining the last sector number for the last partition in the disk image, then cutting off anything preceding the last used sector.
Any data that exists after the last sector will be irrevocably lost. Proceed with caution.
The command to do so will look something like the following:
user $truncate --size=$[(LAST_SECTOR+1)*BLOCK_SIZE] disk.imgDisk sector size can be determined using fdisk or sfdisk:
user $fdisk -l disk.imguser $sfdisk -l disk.imgCompress
To get the best backup-to-space ratio, compressing raw images after the capture will be highly beneficial.
Disk images can be compressed just like any other file. For example, to compress with an "ultra" high compression (--ultra) using 16 threads (-T16) with the zstd compression algorithm (requires app-arch/zstd):
user $zstd --ultra -22 -T16 /path/to/disk.imgdisk.img : 12.50% ( 59.6 GiB => 7.46 GiB, disk.img.zst)
The example above displays a 59.6 GiB disk image compressing down to less than 8 GiBs. Successful compression levels may vary depending on the how much free space is in the disk image, among other variables.
Restore
Observe an extra level of precaution when restoring disk images to a destination block device (such as /dev/mmcblk, /dev/nvme0n, or /dev/sd). The write operation is destructive and one way only, since it removes the target's boot record, partition tables, and filesystem data. Without a disk image backup, the data on the block device will be lost forever.
As long as the block device which will be used for the restoration is large enough to hold the disk image, restoration is possible by writing the disk image directly to the attached block device. This can be performed a number of ways, such as using a dd implementation, like ddrescue:
root #ddrescue -f -n /root/disk.img /dev/sdb /root/disk.img.logCompressed disk images can be restored to a attached block device by decompressing the file and writing the output directly to the block device.
For example, the file disk.img.xz was compressed using the xz utility (from app-arch/xz-utils). xzcat can be used to decompress the file directly onto a new block device:
xz decompression (with progress):
root #xzcat --verbose disk.img.xz > /dev/sdbzstd decompression (with progress):
root #zstdcat --progress disk.img.zstd > /dev/sdbThis has the advantage of not needing to decompress the compressed disk image, which may require more space than the host file system has available.
Troubleshooting
Filesystem image will not mount
Filesystems occasionally have problems. For example, presume in the example below that sda1.img is a btrfs formatted filesystem image and is already attached to /dev/loop0.
root #mount -o loop -t btrfs /dev/loop0 /mnt/imagemount: /mnt/image: can't read superblock on /dev/loop0
Oh, no! Looks like there's a problem with the filesystem image. No to worry, this may be correctable. Filesystem images can be checked just like actual devices.
fat
ext4
btrfs
If the image is just one large btrfs filesystem, the following repair command should work.
root #btrfsck --repair /dev/loop0enabling repair mode WARNING: Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck can successfully repair all types of filesystem corruption. Eg. some software or hardware bugs can fatally damage a volume. The operation will start in 10 seconds. Use Ctrl-C to stop it. 10 9 8 7 6 5 4 3 2 1 Starting repair. Opening filesystem to check... Checking filesystem on /dev/loop0 UUID: 3f345b95-ff66-46b2-b7f5-4a55d8961ea8 [1/7] checking root items Fixed 0 roots. [2/7] checking extents No device size related problem found [3/7] checking free space cache cache and super generation don't match, space cache will be invalidated [4/7] checking fs roots reset isize for dir 792 root 5 reset isize for dir 3426 root 5 reset isize for dir 4895 root 5 reset isize for dir 4940 root 5 reset isize for dir 5561 root 5 reset isize for dir 5571 root 5 reset isize for dir 258862 root 5 Trying to rebuild inode:9223372036855356061 Can't get file name for inode 9223372036855356061, using '9223372036855356061' as fallback Can't get file type for inode 9223372036855356061, using FILE as fallback Moving file '9223372036855356061' to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 9223372036855356061 reset nbytes for ino 9223372036855356061 root 5 reset isize for dir 792 root 819 reset isize for dir 3426 root 819 reset isize for dir 4895 root 819 reset isize for dir 4940 root 819 reset isize for dir 5561 root 819 reset isize for dir 5571 root 819 reset isize for dir 258862 root 819 Trying to rebuild inode:9223372036855356061 Can't get file name for inode 9223372036855356061, using '9223372036855356061' as fallback Can't get file type for inode 9223372036855356061, using FILE as fallback Moving file '9223372036855356061' to 'lost+found' dir since it has no valid backref Fixed the nlink of inode 9223372036855356061 reset nbytes for ino 9223372036855356061 root 819 T[5/7] checking only csums items (without verifying data) [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) found 173687640064 bytes used, no error found total csum bytes: 151170848 total tree bytes: 2374696960 total fs tree bytes: 1938522112 total extent tree bytes: 230211584 btree space waste bytes: 532258492 file data blocks allocated: 3031879557120 referenced 258574393344
Now that the filesystem has been repaired, proceed to mount as normal:
root #mount -o loop -t btrfs /home/larry/sda1.img /mnt/imageYay! The issue has been fixed!
See also
- qemu — a generic, open source hardware emulator and virtualization suite.