Problems Booting From USB Flash

Overview

Unfortunately, there are multiple potential problems when trying to boot from USB flash (and "flash like") devices, mostly caused by historical concerns combined with a lack of adequate standardisation.

History

Ancient Computers

Originally USB didn't exist and no computer had firmware that was able to boot from any USB device.

When USB was created various manufacturers made (ISA and PCI) USB controller cards, and these cards were plugged into computers that don't support USB. In theory, if a standard describing how "boot from USB" was supposed to work existed these cards could have included device ROMs that hook/replace the BIOS "int 0x13" and made it possible to boot from USB, but there was no standard so none of the USB controller cards could do that.

Very Old Computers

Eventually firmware vendors realised it'd be nice to be able to boot from USB. Unfortunately there was still no standard describing how "boot from USB" should work, so different firmware vendors made different choices. Some decided that because USB is a removable device (unlike hard drives at the time) it should act like a floppy disk. Some decided that because USB is (potentially) much larger than a floppy disk it should act like a hard disk. Some decided to support both and provide a setting in their BIOS configuration so that half the time it doesn't work until the end user figures out why and finds/changes the BIOS setting.

Old Computers

It didn't take too long before firmware vendors started inventing schemes to try to auto-detect whether a USB device should behave like floppy or like hard drive. Their intentions were good (trying to avoid end user hassle), but there was no standard describing how "auto-detect how to boot from USB" was supposed to work so their implementations were hideous hacks that were incompatible between firmware vendors.

Some checked for a "valid looking" BIOS Parameter Block (BPB) in the MBR and decided it should behave like a floppy if there was a BPB (and behave like a hard disk if there was no BPB). Some checked for a "valid looking" partition table in the MBR and decided it should behave like a hard disk if there was a partition table (and behave like a floppy disk if there was no partition table). Some checked for a "valid looking" BPB and a "valid looking" partition table and made "unknown" decisions if there was neither or both (possibly including deciding that the device is not bootable when there's no BPB and not partition table).

Modern Computers

El Torito

Some USB flash device vendors have decided that (some) USB flash devices should also be able to emulate a USB CD drive (e.g. as part of the "U3 specification", where firmware sees something that looks like a USB hub with a USB flash device and a USB CD device attached). This adds another layer of complications to the previous "auto-detect how to boot from USB" schemes (deciding if the computer should boot from USB flash or from "same USB flash emulating a CD").

EFI/UEFI Compatibility

When EFI/UEFI was created a new partitioning scheme (GPT) was added and the way an OS boots (for all devices) was changed. For most cases firmware expects to find a FAT volume (e.g. the EFI System Partition) containing a boot loader file and loads the entire file (not just one sector).

However, a lot of computers support BIOS and UEFI; and some of these computers have another layer of "auto-guesswork hackery" to try to decide if a USB device should be booted using UEFI or BIOS. This mostly involves detecting if the device is partitioned using GPT and/or has something that looks like a (FAT) EFI System Partition and/or has a boot loader within that partition.

Other Problems

Firmware Bugs

For some unknown reason; some computers will check for a "valid looking" MBR (with an active/boot partition), and then load that boot sector from within that partition and execute it without ever executing any code in the MBR.

Microsoft Disk Signatures

At some point (starting with Windows NT version 3.5); Microsoft decided they felt like adding a "unique" 32-bit disk signature to the MBR (and keep track of the signatures for every disk that Windows had seen in the registry). They decided to use the 4 bytes at offset 0x01B8 in the MBR for this disk signature, and decided that if the bytes in this location are not "unique" it would be fine for Windows to modify them, even if the device is never used by Windows, even if the device has no partitions that Windows is able to mount, even if it trashes executable MBR code for (older) competing operating systems, even if it destroys TPM checks done by competing operating systems, even though it conflicts with "prior usage" for competing commercial operating systems (PTS-DOS and DR-DOS), and even though (for partitioned devices) the MBR (and any sectors before the first partition) should be considered "owned by a boot manager that is not part of any OS" (and therefore an OS should assume it lacks the authority to tamper with the MBR without explicit consent from a user/admin).

In addition, there is no (known to me) documentation/standard that describes how these "unique" 32-bit signatures are supposed to be generated. This means that when 2 or more operating systems are used on the same computer they can "fight" (disagree on what the signature should be), causing the signature set by one OS to be trashed by another OS. Note that (to avoid problems caused by what I'd call "Microsoft's malicious disregard for other operating systems") Microsoft's disk signature was added to the UEFI specification via. a brief/inadequate description saying that it "may be used by the OS" (but not saying which OS may use it, or how).

Booting Reliability

To increase the chance that an OS on a USB flash device will be booted properly by the BIOS on an unknown computer; the OS has to take into account all of problems described above. This means:

• The MBR must have a partition table with an active partition with the boot loader starting in the active partition (in case the firmware doesn't support "floppy emulation")
• The MBR must have code to "chain-load" the boot loader in the active partition (in case the firmware doesn't fail to execute the MBR's code and decides to treat the device like a hard drive)
• It should have a "DOS 3.31 BPB" (a BIOS Parameter Block including the 4-byte "large total logical sectors" field at offset 0x020) in case the firmware doesn't like the partition table
• The MBR code should be prepared to handle both cases (treated as floppy or treated as hard disk by BIOS)
• The MBR code must not use the 4 bytes at offset 0x01B8 in the MBR for anything
• The device may (optionally) also have El Torito tables to support being treated as a CD (but can't assume that the firmware will use it if the device is USB flash).
• The device may (optionally) also support UEFI such that the device is bootable on both BIOS and UEFI systems (e.g. with a "hybrid GPT" and/or with El Torito tables)

Note: If these guidelines are followed, the MBR's code will be limited to the 404-byte area starting at offset 0x024 and ending at offset 0x01B7 (excluding the initial "jmp" at offset 0x000).