Mac OS X Internals: A Systems Approach

11.4. Disk Images

Exploring and experimenting with disks and file systems is a potentially risky activity in that a mistake might lead to catastrophic data loss. A safer and more convenient alternative is to work with virtual disksor disk imagesrather than physical disks. In the simplest terms, a disk image is a file containing what would normally reside on a physical storage device. Given the appropriate support from the operating system, virtual disks behave just like their physical counterparts. The following are some examples of using virtual disks.

• You could read raw data from a floppy disk and save it to a filesay, using the dd command on Unix or RAWRITE.EXE on Windows. The file so obtained is a block-by-block image of the physical floppy disk. It could be written back to another, similar physical disk to yield a sector-by-sector copy of the original. This process could be used to obtain a disk image from an optical disk, a hard disk partition, or even an entire hard disk.

• Many operating systems allow regular files to be accessed as virtual disk device nodes. These virtual disks may be used like regular disks: You can partition them, use them for swapping, and create file systems on them. Examples include Linux "loop" devices, the Solaris lofi driver, and BSD's vnode pseudo disk (vn) driver.

• Virtualizers and emulators typically use disk images for running guest operating systems.

• A disk image could be compressed or encrypted. Mac OS X's File Vault feature employs an encrypted disk image to provide an encrypted home directory for a user.

Thus, disk images are useful for archives, software distribution, emulation, virtualization, and so on. They are well suited for file system experimentation because they allow you to perform potentially dangerous operations without having to worry about loss of precious data.

11.4.1. Using the hdiutil Program

Apple has used disk images for a long time, primarily for software distribution. The Mac OS X Disk Images framework (System/Library/PrivateFrameworks/DiskImages.framework) is a private framework that provides comprehensive support for disk images. The hdiutil command-line program, which we will frequently use in this chapter and the next, is a versatile tool to access this framework's functionality.

Warning

If you try disk-image-based examples on your computer, be warned that you must be careful about the device node names you use. The I/O Kit dynamically assigns names such as /dev/disk1 and /dev/disk1s2 to devices depending on the number of diskswhether real or virtualcurrently attached. Therefore, if you have multiple real disks attached to your computer, disk1 is likely to refer to a real disk on your system. We will use fictitious disk numbers that start from 10 in our examplesthat is, disk10, disk11, and so on. Please note and use the dynamic names assigned to virtual disks on your system.

The following hdiutil command line creates a file (/tmp/hfsj.dmg) containing a 32MB disk image. It also partitions the resultant virtual disk using the Apple partitioning scheme and creates a journaled HFS Plus file system on the data partition. The resultant volume's name is HFSJ.

$ hdiutil create -size 32m -fs HFSJ -volname HFSJ -verbose /tmp/hfsj.dmg Initializing... Creating... ... DIBackingStoreCreateWithCFURL: creator returned 0 DIDiskImageCreateWithCFURL: creator returned 0 DI_kextWaitQuiet: about to call IOServiceWaitQuiet... DI_kextWaitQuiet: IOServiceWaitQuiet took 0.000013 seconds Formatting... Initialized /dev/rdisk10s2 as a 32 MB HFS Plus volume with a 8192k journal Finishing... created: /tmp/hfsj.dmg hdiutil: create: returning 0

Using the -debug option instead of -verbose causes hdiutil to print excruciatingly detailed progress information. Normally, we will not use either of these two options.

We can mount a disk image in several ways. Double-clicking a disk image's icon in the Finder or opening it with the open command-line utility launches /System/Library/CoreServices/DiskImageMounter.app to handle the mounting. Alternatively, hdiutil can be used to attach the image to the system as a device. hdiutil, along with a helper program (diskimages-helper), communicates with diskarbitrationd to attempt to mount the volumes contained on the disk.

$ hdiutil attach /tmp/hfsj.dmg /dev/disk10 Apple_partition_scheme /dev/disk10s1 Apple_partition_map /dev/disk10s2 Apple_HFS /Volumes/HFSJ

diskimages-helper resides in the Resources directory of the Disk Images framework.

The disk image appears as a virtual disk with /dev/disk10 being its block device node. The pdisk utility can dump partition information from this disk just as in the case of a real disk.

$ pdisk /dev/rdisk10 -dump Partition map (with 512 byte blocks) on '/dev/rdisk10' #: type name length base ( size ) 1: Apple_partition_map Apple 63 @ 1 2: Apple_HFS disk image 65456 @ 64 ( 32.0M) 3: Apple_Free 16 @ 65520 Device block size=512, Number of Blocks=65536 (32.0M)

Detaching a disk unmounts and ejects it.

$ hdiutil detach disk10 "disk10" unmounted. "disk10" ejected.

By default, hdiutil uses a disk image format called Universal Disk Image Format (UDIF). Moreover, the default partition layout contains a partition map with space for 63 map entries, a single data partition of type Apple_HFS, and a trailing free partition containing 16 blocks. This layout is called Single Partition UDIF (SPUD). hdiutil also supports other partition layouts, for example:

$ hdiutil create -size 32m -volname HFSJ_UCD \ -fs HFSJ -layout "UNIVERSAL CD" /tmp/hfsj_ucd.dmg ... $ hdiutil attach /tmp/hfsj_ucd.dmg /dev/disk10 Apple_partition_scheme /dev/disk10s1 Apple_partition_map /dev/disk10s2 Apple_Driver43 /dev/disk10s3 Apple_Driver43_CD /dev/disk10s5 Apple_Driver_ATAPI /dev/disk10s6 Apple_Driver_ATAPI /dev/disk10s7 Apple_Patches /dev/disk10s9 Apple_HFS

In particular, a partition layout of type NONE creates an image with no partition map.

$ hdiutil create -size 32m -volname HFSJ_NONE \ -fs HFSJ -layout NONE /tmp/hfsj_none.dmg ... $ hdiutil attach /tmp/hfsj_none.dmg ... /dev/disk11

Note that a single device entry is listed when the image is attachedthere are no slices. pdisk will not dump partition information in the absence of a partition map, but hdiutil's pmap verb can be used instead.

$ hdiutil pmap /dev/rdisk11 Partition List ## Dev_______ Type_______________ Name_____________ Start___ Size____ End_____ -1 disk11 Apple_HFS Single Volume 0 65536 65535 Legend - ... extended entry + ... converted entry Type 128 partition map detected. Block0.blockSize 0x0200 NativeBlockSize 0x0200 ...

11.4.2. RAM Disks

A memory-backed virtual disk device can be created using hdiutil as follows:

$ hdiutil attach -nomount ram://1024 /dev/disk10

Given ram://N as the device argument, hdiutil creates a RAM disk with N sectors, each 512 bytes. As with disk-backed disk images, we can partition a RAM disk, create file systems on it, and so on.

$ newfs_hfs -v RAMDisk /dev/rdisk10 Initialized /dev/rdisk10 as a 512 KB HFS Plus volume $ mkdir /tmp/RAMDisk $ mount_hfs /dev/disk10 /tmp/RAMDisk $ df /tmp/RAMDisk File system 512-blocks Used Avail Capacity Mounted on /dev/disk10 1024 152 872 15% /private/tmp/RAMDisk

Detaching a RAM disk frees any physical memory associated with it.

$ umount /tmp/RAMDisk $ hdiutil detach disk10

hdiutil and the Disk Images framework have several other interesting features, such as the ability to use disk images specified with HTTP URLs, support for encrypted disk images, and support for shadowing, wherein all writes to an image are redirected to a shadow file (when a read occurs, blocks in the shadow file have precedence).

11.4.3. The BSD Vnode Disk Driver

As we have seen, hdiutil automatically attaches a virtual device node under /dev to a disk image file and prints the dynamically assigned device name. Mac OS X provides another mechanism, the BSD vnode disk driver, which allows files to be treated as disks by attaching them to specific "vn" device nodes. The /usr/libexec/vndevice command-line program is used to control this mechanism.

$ hdiutil create -size 32m -volname HFSJ_VN \ -fs HFSJ -layout NONE /tmp/hfsj_vn.dmg ... $ sudo /usr/libexec/vndevice attach /dev/vn0 /tmp/hfsj_vn.dmg $ mkdir /tmp/mnt $ sudo mount -t hfs /dev/vn0 /tmp/mnt $ df -k /tmp/mnt Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/vn0 32768 8720 24048 27% /private/tmp/mnt $ sudo umount /tmp/mnt $ sudo /usr/libexec/vndevice detach /dev/vn0

11.4.4. Creating a Virtual Disk from Scratch

Although we will mostly use hdiutil to create disk images with automatically constructed partition layouts and file systems, it is instructive to look at an example that starts with a "blank disk." The latter is simply a zero-filled filesay, one created using the mkfile program.

$ mkfile 64m blankhd.dmg

So far, we have seen that attaching a disk image using hdiutil also mounts the volumes within the image. We can instruct hdiutil to only attach the image and not mount any volumes, thereby giving us block and character devices to use as we please.

$ hdiutil attach -nomount /tmp/blankhd.dmg /dev/disk10

Since this disk has no partitions yet, nor even a partition scheme, pdisk will not display any partition information for it. We can initialize a partition map using pdisk.

$ pdisk /dev/rdisk10 -dump pdisk: No valid block 1 on '/dev/rdisk10' $ pdisk /dev/rdisk10 -initialize $ pdisk /dev/rdisk10 -dump Partition map (with 512 byte blocks) on '/dev/rdisk10' #: type name length base ( size ) 1: Apple_partition_map Apple 63 @ 1 2: Apple_Free Extra 131008 @ 64 ( 64.0M) Device block size=512, Number of Blocks=131072 (64.0M) ...

As we saw earlier, in the Apple partitioning scheme, a disk partition map entry is 512 bytes in size. Our partition map occupies 63 blocks, each 512 bytes. Therefore, it has room for 63 map entries. Since the first entry is used for the partition map itself, we have room for as many as 62 new partition map entries, or 61, if the last entry is used for any trailing free space. Let us use, say, 31 of them^[5] to create partitions, even though it is rather uncommon to actually need so many partitions. The following shell script attempts to create 31 partitions, assigning 1MB of disk space to each.

^[5] The Mac OS X 10.4 kernel panics if all 61 entries are used.

#! /bin/zsh # usage: createpartitions.zsh <raw device> DISK=$1 base=64 foreach pnum ({1..31}) pdisk $DISK -createPartition Partition_$pnum Apple_HFS $base 2048 base=$[base + 2048] end

Let us run the script with the name of our blank disk's raw device node as an argument. Note that pdisk prints the number of the map entry it uses while creating a partition.

$ ./createpartitions.zsh /dev/rdisk10 2 3 ... 31 32 $ pdisk /dev/rdisk10 -dump Partition map (with 512 byte blocks) on '/dev/rdisk10' #: type name length base ( size ) 1: Apple_partition_map Apple 63 @ 1 2: Apple_HFS Partition_1 2048 @ 64 ( 1.0M) 3: Apple_HFS Partition_2 2048 @ 2112 ( 1.0M) 4: Apple_HFS Partition_3 2048 @ 4160 ( 1.0M) ... 32: Apple_HFS Partition_31 2048 @ 61504 ( 1.0M) 33: Apple_Free Extra 67520 @ 63552 ( 33.0M) ...

We can now create file systems on each of the Apple_HFS data partitions. Note that at this point, the block and character device nodes corresponding to each partition will already exist in the /dev directory.

#! /bin/zsh # usage: newfs_hfs.zsh <raw device> DISK=$1 foreach slicenum ({2..32}) # first data partition is on the second slice fsnum=$[slicenum - 1] newfs_hfs -v HFS$fsnum "$DISK"s$slicenum end

Again, we run our script with the virtual disk's raw device as an argument.

$ ./newfs_hfs.zsh /dev/rdisk10 Initialized /dev/rdisk10s2 as a 1024 KB HFS Plus volume Initialized /dev/rdisk10s3 as a 1024 KB HFS Plus volume ... Initialized /dev/rdisk10s31 as a 1024 KB HFS Plus volume Initialized /dev/rdisk10s32 as a 1024 KB HFS Plus volume $ hdiutil detach disk10 $ open /tmp/blankhd.dmg ...

At this point, all 31 volumes should be mounted under /Volumes. Detaching the disk will unmount all of them.

Let us also look at an example of creating GUID-based partitions using the gpt command. We will assume that we already have a blank disk image attached, with /dev/rdisk10 being its raw device node.

$ gpt show /dev/rdisk10 # we have nothing on this 64MB disk start size index contents 0 131072 $ gpt create /dev/rdisk10 # create a new (empty) GPT $ gpt show /dev/rdisk10 start size index contents 0 1 PMBR 1 1 Pri GPT header 2 32 Pri GPT table 34 131005 131039 32 Sec GPT table 131071 1 Sec GPT header $ gpt add -s 1024 -t hfs /dev/rdisk10 # add a new partition $ gpt show /dev/rdisk10 start size index contents 0 1 PMBR 1 1 Pri GPT header 2 32 Pri GPT table 34 1024 1 GPT part - 48465300-0000-11AA-AA11-00306543ECAC 1058 129981 131039 32 Sec GPT table 131071 1 Sec GPT header $ gpt add -i 8 -s 1024 -t ufs /dev/rdisk10 # add at index 8 $ gpt show /dev/rdisk10 ... 34 1024 1 GPT part - 48465300-0000-11AA-AA11-00306543ECAC 1058 1024 8 GPT part - 55465300-0000-11AA-AA11-00306543ECAC ...