Is this possible to boot an OS from some kind of a file?

Question

First of all, I want to say that this quesiton is purely theoretical. But if it's possible, I would be really glad to learn its possible use cases, and overall drawbacks.

Is this possible to boot an OS, let's say a Linux, from a file, rather then a drive? Like a file that has read/write access on a micro-USB. Imagine that you plugged in your USB. And chose the option that makes you boot from it. Now, there is a tiny-OS-like-software on this USB that controls booting, like grub. There are two primary partitions. The first one is for this micro-OS, and the second partition is like FAT, for data storage. And also, the latter one has multiple files as operating systems.

When system boots, this tiny-booting-OS reads the contents of one of those files, loads the kernel into the memory, and the kernel accesses the file, like it was accessing a drive. This file has different segments as directories and files, and stuff.

In practice, I guess I am thinking of something like a file-based filesystem. Do such a thing exist? If so, I think that it is either an ancient tech or a really unpractical one. Or maybe not? Would like to learn more about it.

Answer 1

Yes, that's more or less how every Linux "live USB" system works. They all boot from a read-only filesystem image (often a .squashfs file).

The bootloader is not really involved (it gets out of the way as quickly as it can); the OS kernel handles everything.

(It's also how Windows install USBs work – the mini OS really boots from a Boot.wim archive.)

In practice, I guess I am thinking of something like a file-based filesystem. Do such a thing exist?

Any filesystem can be put in a file, as long as the OS knows generally how to attach a filesystem that's stored in a file. For example, you might have seen this done with an .iso file (or a .dmg on macOS). And vice versa, any file format can be called a "file-based filesystem" as long as the OS knows how to present its contents as files. (You could almost say that .zip archives are a file-based filesystem...)

When system boots, this tiny-booting-OS reads the contents of one of those files, loads the kernel into the memory, and the kernel accesses the file, like it was accessing a drive. This file has different segments as directories and files, and stuff.

Yes, many operating systems have a feature that can attach files as if they were virtual disks which behave the same as a real disk as far as filesystems are concerned, so that you can partition them and/or place any regular filesystem in them. Although this isn't always usable for booting the OS, but it's fairly common as far as the ability to store a whole filesystem within a file goes.

• Linux allows setting up "loop" devices as virtual disks corresponding to a file. The device can then be partitioned or directly formatted.

• OpenBSD and NetBSD have the same function in "vnd" (vnode disk) devices; FreeBSD has "md" devices; Illumos (Solaris) has "lofi" devices; OpenVMS has "LD" devices...

• I'm not sure what macOS has, exactly, but it's how .dmg files work – they're not archives; they're compressed filesystem images which macOS attaches as virtual disks.

• Windows doesn't seem to have a fully generic feature, but as of Win10 it does allow attaching .iso and .vhd files. Double-click an .iso file (which typically an ISO-9660 filesystem, hence the name) and it'll be attached as a CD-ROM. If you open DiskMgmt, you can create a .vhd file that can be partitioned, formatted with NTFS, and so on.

  (Before Windows 10, plenty of third-party tools existed that did the same thing… mainly for CDs though – even back in Win98 days, you'd have software like Daemon Tools or Alcohol 120% to create virtual CD drives for your Legally Acquired Games – but virtual HDDs were certainly a thing as well.)

Of course, not all of these can be used for booting the whole OS – in most cases you need to have the kernel already running and some drivers loaded in order to access the filesystem in which the image is contained.

For Linux specifically, an initramfs is used to solve that problem; it's a tiny filesystem archive that's the first thing to start after the kernel (containing the initial drivers) and contains whatever tools are necessary to mount the real / filesystem. The actual kernel and the initramfs – two files in total – must still live in a location that the bootloader can access (although some bootloaders such as GRUB do support loading them out of a filesystem image as well), but once those are loaded into memory, the initramfs can set up any kind of environment it needs.

Most Linux distributions opt to use an initramfs even for basic disk and filesystem drivers, but it's a fairly small change to make it set up a loop device (or just about anything else) as an additional step. For example:

• Nearly all Linux CDs/USBs consist mainly of a file containing a special SquashFS filesystem. (This is actually rather close to being a "file-based filesystem"; it is meant specifically for this kind of read-only usage, with its internal layout designed to pack everything closely together, unlike traditional filesystems that do the opposite.)

  You could easily extend this to have several boot menu items, each of which would load a different initramfs and/or specify a different rootfs image as needed.

• Some "live CD" builds have a "persistent" mode where a second filesystem image is attached, containing a more traditional writable filesystem such as Ext4, as an overlay to store all changes made to the live environment.

• Of course, a regular Ext4 image could be used directly as well. A few years ago, Ubuntu used to have the "Wubi" project which would do a one-click install of the full Ubuntu system into an Ext4 image stored on your Windows NTFS partition.

  The bootloader (Grub4dos) had sufficient knowledge of NTFS to find the kernel and the initramfs, while the initramfs then loaded the ntfs-3g driver and set up a 'loop' disk device.

• There indeed exist programs that can build you a "multi-Linux" USB stick, at least some of which could boot .iso images dropped into the USB stick's regular filesystem, making the whole "initramfs/loop" process happen twice in a row.

---

As a completely different example, MS-DOS used to have a "DriveSpace" component which would compress your disk contents; this was also implemented through a driver that stored your FAT filesystem inside a special image file, and during boot would replace the real C:\ with the one stored within the virtual disk.

Answer 2

Yes, this is entirely possible, and the following is one way of doing it.

In Windows, you could have an entire disk with its file-system stored in a .vhdx or a .vhd file.

Then you can use diskpart to attach this file as a virtual disk and assign it a drive-letter.

Finally, you may use bcdboot to assign this virtual disk as the boot disk.

For precise instructions see the Microsoft article
Boot to a virtual hard disk: Add a VHDX or VHD to the boot menu.

Answer 3

You might find the BeOS Personal Edition interesting. Released in 2000, this was a free version of the BeOS operating system that installed on a Windows 95/98 system. The OS image itself was simply a single "image.be" file. All you had to do was double-click an icon and the system rebooted into BeOS. No need to create partitions or modify bootloaders. Windows wasn't even aware that another operating system existed. When you're done, delete that file and all traces of it were gone.

There's not a lot of data on exactly how this worked under the hood but from what I can gather, the file was treated as if it were a partition. The launcher figured out the image file's block address on the hard drive, and then told the BIOS to boot into that address as if it were the start of a hard drive partition. The contents of the file were formatted like a partition, so the BIOS couldn't tell any different.

Bear in mind, this was almost a quarter century ago. Modern OSes, hardware, and EFI subsystems have a lot more security that might make Be's exact technique infeasible. The basic concept is still true, though. All you need to boot is the starting block address of a (contiguous) partition. The low-level bootloader doesn't examine the rest of the disk in detail, so it won't notice or care that it's actually inside another partition.

Answer 4

Every OS already boots from 'a file', which tells it what other things it needs to look for to continue the process. At boot, a computer is only just smart enough to know what the first thing it needs is. Nothing more. If you mean boot from something like an ISO file, then no - because it doesn't know what one is at that point, nor how to read it.
To over-simplify, grub works by loading just enough of an OS to be able to then direct to which file to look at next, to continue booting.

You could do this from a running OS using Virtual Machines. You can then have folders full of 'files' [each really a container format, similar to an ISO] & choose which to boot.

Answer 5

Other answers describe what you probably were thinking of. This answer deliberately describes some concepts in a way you most likely did not think of.

Consider a different view; consider these facts:

• In Linux, devices (like a block device referred to as /dev/sda) and partitions (e.g. /dev/sda1) are also files. They are block special files.

• A block device that contains partitions and a partition table actually contains a simple filesystem. We don't call it "filesystem", but conceptually it is one. Look at it this way:

  • The partition table itself is metadata, the structure of the filesystem that describes files within.

  • Partitions are files.

  • There are many types of partition tables, like there are many types of "real" filesystems. Nowadays GUID Partition Table and dos-type in MBR are common types of partition tables. Some types can store more metadata than others, but in general…

  • … like in a "real" filesystems partitions (i.e. files) have numbers (somewhat like inode numbers), labels (like names), types, flags (like attributes). All this is written in the partition table.

  • And most importantly the partition table contains enough information to tell which logical sectors belong to which partition; and which are "free space" (similarly any "real" filesystem associates blocks to files or marks them as free). There are limitations, differences:

    • Each partition must be continuous (in most modern "real" filesystems files may be fragmented).
    • The size of a partition must be a multiple of the logical sector size of the device (in "real" filesystems the unit is one byte).
    • There is no hierarchy, no directories. It's true that in a dos partition table there can be at most one extended partition containing logical partitions, so it looks like a directory containing files; but it's just an extension allowing us to create more than four partitions the original standard was limited to, you still cannot create "directories" freely. (In modern "real" filesystems you can create directories and subdirectories, but early filesystems (e.g. CP/M) were not hierarchical and directories had yet to be invented; a filesystem without directories is still a filesystem.)

  • Software (like UEFI or the kernel of an OS) has to read the partition table (metadata) and interpret it in order to properly access individual partitions (files). If it lacks support for some type of partition table (filesystem) then it won't be able to perceive its partitions (files) as separate entities. If it has support then you can read from or write to each partition (file) separately. There's a difference though:

    • Modifying the partition table itself is quite cumbersome and low-level in comparison to modifying metadata of files inside a "real" filesystem. There are specialized tools like fdisk or gdisk. You cannot create a partition with touch, you cannot resize a partition with truncate, you cannot actually remove it with rm (removing /dev/sda1 removes the special file but not the actual entry from the partition table).

Despite the differences the concept is the same: a large file contains an inner structure that describes which other parts of the file build what files within.

A filesystem containing partitions and a partition table is designed to hold "real" filesystems as its files and it usually does.

According to this broad understanding of files and filesystems, almost every installation of Linux or Windows (or whatever) uses a "file-based filesystem" (or few of them). The file here is a partition, it holds an inner filesystem, while itself living inside an outer filesystem whose metadata is the partition table.

I know this is probably not exactly what you were thinking of. My point is the whole idea is already widely implemented, we just rarely perceive the implementation this way; and what you probably were thinking of is not something different or new, it's just one more layer in a stack of filesystems.

Answer 6

grub2 can boot from any ISO file that are setup as Casper bootable. The default Live ISO Ubuntu is Casper-compatible ISO, for example, and I've actually done this a couple times to have grub boot Ubuntu from an ISO just sitting in the filesystem without ever the ISO to a CD or extracting them to a USB drive.

Some docs on this: https://help.ubuntu.com/community/Grub2/ISOBoot

Answer 7

For "completeness", a simple way to boot ISO files on a Linux

Ref:
https://wiki.grml.org/doku.php?id=rescueboot

$ sudo apt-get install grml-rescueboot (Debian, Ubuntu)

Select, Download and place a bootable ISO,
e.g. one of the Ubuntu Install ISO:s,
in /boot/grml/ and then do

$ sudo update-grub

Reboot,
Hit ESC as the "BIOS"-texts appear
and you should have access to booting alternatives, including one for the added ISO.

As the alternative OS has booted, you will be able to access all hardware, with the normal Linux methods; using i.e. mount, umount and all that stuff in between those.


WARNING: Not checked / verified for completeness, I have it installed, instructions under Ref replicated here.