What is the purpose of these overlapping partitions?

Question

I want to install Parabola on one of my machines, so I downloaded the .iso, checked the signature and hash, and then flashed the image to a USB thumb drive like so:

dd if=parabola-dual-openrc-cli-2020.01.18-netinstall.iso of=/dev/sde oflag=direct bs=1M status=progress

This seems to work as intended, as I can now boot from this drive, but I discovered something odd. fdisk reports two partitions on the drive now, the first is apparently of an unknown type, and the second one is nested inside it:

$ fdisk -l /dev/sde | grep -i dev
Disk /dev/sde: 7,5 GiB, 8086618112 bytes, 15794176 sectors
Device     Boot Start     End Sectors  Size Id Type
/dev/sde1  *        0 1796095 1796096  877M  0 Empty
/dev/sde2         172  131243  131072   64M ef EFI (FAT-12/16/32)

parted doesn't even "see" sde1, it only shows the second partition.

$ parted /dev/sde -s print | tail -n +7
Number  Start   End     Size    Type     File system  Flags
 2      88,1kB  67,2MB  67,1MB  primary  fat16        esp

GParted goes a step further and only shows the block device as one whole partition of type iso9660.

Now this makes a bit of sense, ISO9960 is apparently the standard defining how data is layed out on optical disks, so when writing an iso-image to a drive 1:1, this seems like a plausible result.

I can indeed mount both of those drives and see all their contents. Only sde1 is mounted read-only, even though optical discs would normally always be read-only.

$ cd /mnt; sudo mkdir new; sudo mkdir new2
$ sudo mount /dev/sde1 new
mount: /mnt/new: WARNING: device write-protected, mounted read-only.
$ sudo mount /dev/sde2 new2

It also seems like most files on sde2 - which only has the directory EFI on it - are also on sde1 in the same location:

$ paste <(tree /mnt/new2/EFI) <(tree /mnt/new/EFI) | column -s $'\t' -t
/mnt/new2/EFI                        /mnt/new/EFI
├── boot                             ├── boot
│   ├── bootx64.efi                  │   ├── bootx64.efi
│   ├── entries                      │   ├── entries
│   │   ├── parabolaiso-x86_64.conf  │   │   ├── parabolaiso-x86_64.conf
│   │   └── uefi-shell-x86_64.conf   │   │   └── uefi-shell-x86_64.conf
│   ├── HashTool.efi                 │   ├── HashTool.efi
│   ├── loader.efi                   │   ├── loader.efi
│   └── refind.conf                  │   └── refind.conf
├── parabolaiso                      ├── parabolaiso
│   ├── parabolaiso.img              │   └── efiboot.img
│   └── vmlinuz.efi                  ├── shellx64_v1.efi
├── shellx64_v1.efi                  └── shellx64_v2.efi
└── shellx64_v2.efi                  3 directories, 9 files
3 directories, 10 files

Going deeper, as these partitions are mounted now, I had a look what options were detected for them:

$ cat /proc/mounts | grep /dev/sde
/dev/sde1 /mnt/new iso9660 ro,relatime,nojoliet,check=s,map=n,blocksize=2048 0 0
/dev/sde2 /mnt/new2 vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0

So sde2 is not just some FAT that seems to be specifically optimised for EFI, but it also has the VFAT extension to allow for longer file names.

What the hell is going on here? Is this how burned iso-images always look, or is Parabola doing something weird and special here?

Is this maybe related to this image being bootable on both i686 and x86_64 architectures?

Answer 1

What the hell is going on here? Is this how burned iso-images always look, or is Parabola doing something weird and special here?

"Burned ISO images" do not change their appearance; the whole purpose of a disk image is that it's a 1:1 representation of the disk and would be written as-is. So this is also what the original .iso file looks like.

The original ISO file looks like this because it has been specifically prepared to be bootable as both a CD/DVD (the original target of ISO images) and a HDD/USB disk. That results in a whole mess because:

• EFI may want a USB disk to have a valid partition table, but CDs and DVDs have no typical HDD partition table at all – neither MBR at sector 0, nor GPT at sector 1. The entire disc is one ISO 9660 filesystem (or UDF or both), with a "Volume Descriptor Set" at sector 16.

• BIOS would want a USB disk to have a boot sector, but CDs don't use that. Instead, they have an El Torito "boot catalog" with multiple entries for different platforms, each of which may point to a different "disk image" with its own boot sector.

  In other words, the ISO 9660 structure may have a whole tiny FDD/HDD image embedded in it so that a BIOS may boot that and pretend it's booting from FDD; and this also extends to EFI-based boot, where the boot catalog may have an entry for "EFI (x86_64)" that points to an image of an HDD containing an EFI System Partition.

  (If you try to open the ISO image with 7-zip, these will be shown in a fake directory named "[BOOT]".)

So if you just took a standard ISO 9660 image and wrote it to a USB stick, it would probably not boot at all, because the BIOS boot sector or the EFI partition table are somewhere deep and not simply starting from sector 0.

As it happens, however, the initial few sectors of ISO 9660 images are "unused" (as some early platforms did put a boot sector in there), so there's a tool called isohybrid that crafts a BIOS boot sector and a partition table for EFI that both point to the El Torito boot images inside the ISO 9660 structures, and once that's done the image has two or three different data structures depending on where you start looking.

• If you burn the image to a CD, then the firmware starts by looking for an ISO 9660 "Volume Descriptor" at sector 16, where it will indeed find that the entire disk is an ISO 9660 filesystem (which has an El Torito boot record containing an embedded FAT32 filesystem).

• But if you have a USB stick, then the firmware may start by looking for a GPT partition table at sector 1, where it will indeed find that it's a GPT-partitioned disk with an 'EFI System Partition' at a slightly weird location.

  (So if you mount that partition and update it, you're technically editing the innards of the normally-read-only ISO 9660 filesystem.)

  But if the GPT only references that single partition, then the rest of the ISO 9660 structures show up as "free space" and are at risk of being damaged if the user creates their own partitions on the same disk, thinking they've got plenty of space.

  To avoid that, isohybrid creates this second partition that encompasses the entire ISO 9660 filesystem – which, naturally, overlaps the EFI partition that was "cut out" from the very same ISO 9660 filesystem.

• And to make things more interesting, if the firmware or software starts by looking for a MBR partition table at sector 0, it will find one as well – with entries that exactly mirror the GPT partition entries.

  Except isohybrid uses the partition type 00 to designate an "empty" partition, and it seems that libparted doesn't quite understand that and thinks that the whole partition slot is empty.

The UEFI specification defines a specific check order for all removable media: 1) look for GPT partitions; 2) look for El Torito; 3) look for MBR partitions. But that doesn't apply to partitioning tools – in your case fdisk finds the MBR first, etc.

To add to the complexity, that the partition discovery done by the kernel is completely separate from partition discovery done by fdisk, gdisk, parted, or GParted. Even though fdisk says that the 1st MBR partition is "/dev/sda1", that has nothing to do with what the kernel thinks /dev/sda1 actually is.

All of these tools have their own code to parse partition tables independently from the kernel. They just assume that the disk's data structures are sensible such that different programs would come to the same conclusion regarding what /dev/sda1 is, but as you can see an 'isohybrid' ISO image is the opposite of that.

So in conclusion, the weirdness comes from trying to make a CD image usable and/or bootable as an HDD image, which is like if you tried to make an .mp3 that's also a valid .zip file.

It all just happens to work because none of the important data structures conflict with each other:

• When the disk is detected as ISO 9660 via sector 16, the code ignores what's in sectors 0–15 because that's part of the "unused" area for ISO 9660.

• When the disk is detected as MBR via sector 0, the code ignores what's in sector 16 because the MBR doesn't extend that far.

• When the disk is detected as GPT via sector 1, the code ignores what's in sector 16 because the GPT (which ends at sector 63) also doesn't extend that far…?

  That last part works because the sector size is different (CDs have a sector size of 2048 bytes while USB sticks have 512-byte sectors), so the Volume Descriptor Set isn't actually at sector 16, it's at sector 64, and the GPT really doesn't extend that far.

• Sector size should also explain why UEFI won't simply detect the disk as ISO 9660 after you write it to a USB stick, despite my earlier mention that it's in the spec for all kinds of media – the firmware will be looking at "sector 16" for that, but because that's now sector 16 in 512-byte units, not in 2048-byte units, it will be looking far away from the ISO 9660 VDS; all it'll find is the empty space in the middle of the GPT.

  And vice versa, when the image is on a CD, then its GPT no longer begins at "sector 1" – it's in the middle of sector 0 now, and therefore won't be detected on CDs.

  Finally, the MBR at sector 0 is still at sector 0, so it would be detected… but because its partition references are also in terms of sectors, they would be completely off. That's why UEFI prioritizes ISO9660 before MBR.

So sde2 is not just some FAT that seems to be specifically optimised for EFI, but it also has the VFAT extension to allow for longer file names.

Those all are the exact same thing; the UEFI specification defines the "EFI file system" as being based on FAT12/16/32 with the LFN extension. ('vfat' is how Linux calls FAT with LFNs as introduced by Windows 95.)

• EFI encompasses the use of FAT32 for a system partition, and FAT12 or FAT16 for removable media. The FAT32 system partition is identified by an OSType value other than that used to identify previous versions of FAT. This unique partition type distinguishes an EFI defined file system from a normal FAT file system. The file system supported by EFI includes support for long file names.

  (UEFI spec. section 13.3)

Answer 2

This is not completely an answer to the whole quenstion since I dont have the deep understanding, but at least some links for future readers who google this.

It's not unique to Parabola, I saw the same in the Debian (live debian 12) image. The post in the debian forum confirmed my observation that gparted and gdisk refuse to meddle with those overlapping partitions of the hybrid-ISO "file system", but fdisk can create a new partition in the free space (though it prints a caution/warning in red).

And wikipedia writes about this hybrid filesystem layout that seems to be quite old

[...] Companies that released products for both DOS (later Windows) and the classic Mac OS (later macOS) could release a CD containing software for both, natively readable on either system. Data files can even be shared by both partitions, while keeping the platform specific data separate. In a "true" (or "shared") hybrid HFS filesystem, files common to both the ISO 9660 and HFS partitions are stored only once, with the ISO 9660 partition pointing to file content in the HFS area (or vice versa) [...]

Here, apparently, the contents of the EFI partition should be readable by the motherboard when booting and also be part of the \ tree later, and they didn't want to do it it with mounting the other FS after startup. I assume, this is neccessary/beneficial for the usage as live CD/USB...?