What actually is "unallocated space"?

Question

Once again, here I am trying to wrap my head around some HDD and filesystem concepts.

When Windows displays a drive as having "unallocated space" and prompts for the drive to be initialised, what exactly does this mean - that the drive has no filesystem on it, or just that the filesystem isn't recognised by Windows?

Is the concept of unallocated space a Windows one, or a hard drive one - in other words, is it possible for other operating systems to write to unallocated space?

If so, does this mean that outside of Windows, a filesystem isn't necessary to write to a hard drive?

Answer 1

is it possible for other operating systems to write to unallocated space?

Yes, any OS can always write to any disk location, whether it's inside or outside a partition.

Partitions are nothing more than a list of regions that the OS agrees to use – the disk always behaves as a single continuous storage medium and doesn't know care about being partitioned.

(Indeed the partition table itself is stored on the same disk, on sectors 0–32ish, and when tools like fdisk or diskpart edit partitions, all they do is update the table stored in those sectors.)

does this mean that outside of Windows, a filesystem isn't necessary to write to a hard drive?

It's not strictly necessary for most operating systems. A program with sufficient rights can open the raw device file such as /dev/sda4 on Linux or \\.\Device\Harddisk5\DR5 on Windows and read/write bytes into any location.

The program could practically invent its own private filesystem that way, or use some other structure (e.g. there are SQL database engines which write the DB into a raw partition). Or it could just dump data wherever.

Filesystems as you know them merely allow all programs within the OS to share the same drive or partition. The OS takes care of space allocation, reliability, etc. and programs don't need to think about anything other than files and folders.

When Windows displays a drive as having "unallocated space" and prompts for the drive to be initialised, what exactly does this mean - that the drive has no filesystem on it, or just that the filesystem isn't recognised by Windows?

When the "Disk Management" app prompts for the drive to be initialised, that means it does not even recognize the partition table (which is supposed to indicate which areas are allocated and which are not).

Windows understands MBR and GPT partition tables, but when this prompt shows up, it means neither was found. Therefore Windows doesn't even get to the point of looking at filesystems – it cannot know where the filesystems are (if there are any).

However, when Windows finds a partition with unrecognized filesystem, "Disk Management" does not prompt to format or otherwise overwrite it – it simply shows that partition as having an unrecognized filesystem, often shortened as "RAW".

When other operating systems create their partitions on an MBR or GPT disk, they will generally honor the existing partition table and won't start using a different one. So if e.g. Linux or FreeBSD created its own partitions, they will be visible as actual partitions in Disk Management – as long as they're in the MBR or GPT, they won't simply turn into "unallocated space" just because the filesystem was not recognized.

However, do not confuse DiskMgmt's prompt to initialize the disk with Windows Explorer's prompt to format the disk. The former is about unknown partition tables, but the latter is about unknown filesystems in a known partition.

(Sometimes Windows accidentally assigns a drive letter to an unknown filesystem, and trying to open it via Windows Explorer will cause a "Format" prompt.)

So in short:

• Prompt to initialize disk ⇒ No partition table (MBR or GPT) was found. The disk might be empty or it might use a non-standard partition table such as APM.

• No prompt, but unallocated space shown ⇒ The space is really unallocated. The disk's partition table has no entries that would correspond to that area.

• No prompt, but unknown partition shown ⇒ The partition contains an unrecognized filesystem.

• Prompt to format disk ⇒ The partition contains an unrecognized filesystem.

Answer 2

Let's first explain how a harddrive works.

A harddrive has space, obviously. You create a partition scheme across the entire drive, and you fill that with partitions. A partition is a reservation in the partition space with a fixed size. (this is a very basic explanation. If I add RAID to the mix, I fear the answer gets unnecessary complex)

If your harddrive has just one partition, and its size is smaller than the size of the harddisk, there is space left for another partition. This bit is called unallocated space.

Now, does this mean you can always create a partition there? No. It is possible that another OS created a partition in that space that is completely unfamiliar by your OS.

If this is the case, your OS will either see this as an unknown partition, or in the worst case as unallocated space. If you create a partition in unallocated space that is actually allocated by another OS, you will destroy the previous partition in most cases.

There is another possibility, the drive has a partition scheme that is not familiar by your OS. If this is the case, the OS will think the drive is not initialized, and will ask you to initialize it. In Windows, it will either be GPT or MBR, but there are known cases where the drive (or SD Card etc) is formatted for a special device and its unique format is a form of security.

Answer 3

A hard drive (or its equivalent such as a USB thumb drive) by convention has a partition table on it. An MBR (master boot record) style partition table fits in one sector (512 bytes) at the beginning of the drive (sector number 0). An MBR also contains a boot loader so the partition table space is limited to four "primary" partitions (though some 1980s software supported eight partitions). Each MBR partition entry has:

• a type code (a single byte) indicating that the corresponding partition is either NTFS, FAT-32 (four flavors), FAT-16 (eight flavors), FAT-12, Linux swap area, etc. Partition code 0 (zero) means "raw", or "no meaningful content", "or disregard partition contents".
• whether the partition is flagged as bootable. Up to one partition per disk can have this flag enabled.
• The disk address of the beginning sector of the partition
• The partition's ending sector—or the number of sectors. ^{Early PCs used cylinder/head/sector numbers; after about 1995, use of "logical block numbers" a single 32-bit value replaced cyl/head/sect addressing}

Since a maximum of four partitions was soon intolerable, a mechanism for additional partitions was "extended partitions" in MBR where a specially marked partition can have an embedded MBR containing another partition table. Thus it is possible to divide a disk into dozens of partitions though some partitioning tools might inflict some artificial limit.

A GPT is intrinsically not limited to four partitions. Its basics occupy more than 512 bytes on disk, usually something in the 4 to 16 megabyte range. But the meaning of partitions is the same.

---

A disk's partition table which has every sector of the disk included in it has "no unallocated space" by the most common use of the phrase. So the usual answer to your question is that unallocated space is that outside of all partitions. However, ambiguity of terminology occurs if a partition is marked as type zero (raw), so there possibly could exist a tool which considers a raw partition as, in some sense, unallocated.

When Windows displays a drive as having "unallocated space" and prompts for the drive to be initialised, what exactly does this mean - that the drive has no filesystem on it, or just that the filesystem isn't recognised by Windows?

It means there is no partition table or all the partition entries are empty. Windows is happy to ignore unrecognized filesystems.

Is the concept of unallocated space a Windows one, or a hard drive one - in other words, is it possible for other operating systems to write to unallocated space?

It is a hard drive convention observed by all standard operating systems supporting PC software. Not sure what "in other words" means. Yes, it is possible to write to unallocated space (even in Windows), but the proper etiquette is to create a partition containing the unallocated space, format a filesystem within it (which writes to the "unallocated space"), and then write files and directories to it.

If so, does this mean that outside of Windows, a filesystem isn't necessary to write to a hard drive?

For a well behaved system, no. Cooperative systems require filesystems to be in place, permissions set appropriately, etc. However, it would be trivial in Linux (for example) to open the disk itself as a big file (raw disk access) and write directly to it. It would require administrative access and some guidance or calculations to determine where to write.

However, ignoring filesystems and partitions is bad. They greatly improve disk management and make life much easier and more sane. Ultimately it is possible to write renegade software which completely ignores sanity and good manners and writes anywhere on the disk it wants.

However, the meta-ness I perceive from this question might be misunderstanding whether you intend to obey the permissions and limits imposed by non-administrator/non-system mode in Windows, Linux, etc.

Answer 4

Other answers already tell a great deal, I'll just add a bit more details to fill some gaps.

One of the question the author asks is:

If so, does this mean that outside of Windows, a filesystem isn't necessary to write to a hard drive?

Actually, you don't need it even inside Windows. But the devil is in the details:

At a low level, the hard drive is just a bunch of bytes that can be read and written. Each byte has a number - the 1st byte, the 2nd byte, etc. And... that's it. Just a bunch of numbered bytes. Note that there's no concept of a "file" or "directory" or "partition" or whatever. When an operating system wants to read or write to the drive, at the lowest level it just says to it "please give me bytes 123 to 456" or "please write this data into bytes 789 to 1234". Or something like that.

In fact, you can easily write a program like that too. If you're a programmer, all you need to do is open a special filename \\.\PhysicalDrive0 like a regular file and you get full access to ALL the bytes on the drive, just as they are written. (Note: this is under Windows. Under Linux you open one of the files under /dev and other OSes might have other mechanisms still)

Mind you, you'll need admin privileges for that, since you do get access to all the sensitive information on the drive. But other than that, that's all there is to it. I've done it myself.

Now, actually, I've lied to you a bit. A little white lie. The drive doesn't actually deal in individual bytes. It reads and writes data in "sectors". That's just a fancy name for "a bunch of bytes at a time". For the "spinning rust" kind of drives, the sectors are usually 512 bytes large. So when you read or write to the drive, you need to do it in 512 byte large chunks. So, Windows actually tells the drive "get me sectors 1 to 5" or "write to sectors 6 to 8". But it doesn't really change the whole idea.

SSDs usually have larger sectors, and the sizes can vary between drives. Typically it's 4096 bytes, but there are drives with 2048 byte sectors, and probably other values too. Again - doesn't really matter, just something to keep in mind. The OS usually abstracts it away and allows you to read and write whatever, and then does a little extra work to make sure the result is what you expect it to be.

Next step are the "partition tables" that wallyk has nicely explained in his answer.

The partition tables aren't anything that the drive cares about - all it cares about is reading and writing sectors of bytes. But we, humans, would like to split the drive into "partitions" and that's where the partition tables come in. A partition table is, by convention, stored at the very start of the drive. It's probably documented in some standard somewhere too, so that all the tools and OSes have the same understanding of them.

The MBR (Master Boot Record) table is an older standard, there since 1982. Since at that time nobody had even dreamed of SSDs or sectors different than 512 bytes, the MBR was designed to take up exactly 1 sector (512 bytes) and had a bunch of other limitations, particularly relating to larger drive sizes.

Eventually the drives got large enough that MBR just wasn't enough, and GPT (GUID Partition Table) was introduced to get around those limits.

Now, whichever format you use, all that the partition table actually says is something along the lines of "The partion 1 exists on sectors 50 to 5000 and is formatted with the FAT32 filesystem; The partition 2 exists on sectors 5001 to 10000 and is formatted with the EXT4 filesystem" and so on.

From this the OS can find the actual partitions and parse the filesystems. And that's where you get files and folders and permissions and other stuff that the OS provides you with. This is all done in software.

And here's another thing to note - as I said, the partition tables are a convention, not a necessity. Hardware doesn't care about it, software does. So you can use a drive even without one. For example, this day and age, it's common practice to use LVM in Linux to manage your hard drives (it's a bit of a long story). And one possibility with that is to add a whole drive to the pool of devices managed by LVM. In this case, there will be no partition table on the drive - LVM will manage it instead.

Answer 5

Imagine a HDD like a magnetic board for kids

When you get a board the first time, it will not be completely white, but a scrambled mix of black and white nonsense. Before you can read or write something, you will first have to erase a part of the board, where you want to write. And since a hard-drive can store all kind of data, you will also have to create some initial information (usually right at the beginning), which defines what kind of data will be on the drive.

I could for example write a pattern of dots in the top left corner to designate the content of the board to be written English. Or another pattern, which would indicate an image. Or a complex pattern, which would divide the board into four parts and defining that the top left quarter will contain an image, the top right will contain English text and the two in the lower half are empty and ready for new data. (And it is enough to write in the pattern "the lower half is empty" and I don't have to clean this part completely, since no one will try to read the content of this space, because it is designated as "empty")

In this way the OS will usually write a clear pattern in the beginning of a drive, which can be used to identify what other data is written on the drive and where it can be found. These parts are the Partition-Table and File System Descriptor.

Unknown data is not easily distinguishable from noise

When an OS encounters an HDD it usually tries to read the first bytes on the drive to identify a certain pattern. Most operating systems know a bunch of different formats how a drive can be partitioned and it tries to find the known markers (Master Boot Record, Partition Tables,...) to identify the contents of the drive. - If an OS (like Windows) can not find a pattern which it recognizes, it does not know, if the drive is just "empty" (filled with random noise, like an magnetic board after shaking it) or if it is just in a format it does not know.

So the "unallocated space" is just unrecognizable data with no identifiable pattern for the OS. If you choose to create a partition in this space, the OS will simply overwrite the first few bytes and create a pattern of data, which essentially says "This drive will contain files in this structure and currently everything after this note is currently empty and can be overwritten" and after that all files which you copy onto the drive will be written in the structure which is defined in this note.

Special cases of "unallocated space"

Unrecognizable space on an HDD can be an unknown file-system. Or it could be a hidden partition generated by an encryption software like VeraCrypt - which is designed so that the hidden encrypted data looks just like random noise on an unallocated disk. Another case can be a damaged file-system. If the hard-drive experienced some damage the pattern in the beginning could be broken, so the system can no longer recognize the file-system, even if it is of a known type. In this case you would need some recovery software, which tries to reconstruct the information about the file-system, so the original data can be accessed again.

Tools to access raw data

There are also tools for Windows/Linux, which allow you to access the raw data of an HDD. These tools will not care if a drive is partitioned or contains just random noise, if you use these you can view the drive in its entirety, just like the magnetic board. There you will see a giant amount of random-looking numbers and symbols, which are usually interpreted by a file-system-driver in the OS to allow you to see a structure of files and folders.

If you are interested in how file-systems work, you could look at a drive with such tools and see the partition tables for yourself.