Doesn't metadata occupy any size?

Question

I put 4096 characters in a text file and save it. Since every character is 1 byte, the Size of the text file must be 4K bytes. As you see below that's OK:

I connect my flash memory to my computer. the free space on the flash memory is 1,717,518,336 bytes:

I created a copy of the file in my flash memory. And again take a look at the free space. it has 1,717,514,240 bytes free space:

Let see what is the difference:

1,717,518,336 - 1,717,514,240 = 4096 bytes

---

My question:

Q1:

As you see in the last picture above, the only space that the file occupy on the flash, is the space for its contents [characters].So where is metadata file?

I mean, when I move the file to another computer, how it understand name of file, Owner of file, Date created and modified and ... ?

Doesn't it occupy any size?!!

Q2:

Can I see the metadata file in the flash memory?

Appreciate your time and consideration.

Answer 1

Yes, metadata occupies space. On NTFS it occupies 1024 bytes, to be specific. However, the information is not stored in the file, but in the master file table MFT. Specifically in MFT record #4 $AttrDef.

See this Technet article for details: table 3.5 holds all MFT records defined.

When a volume is formatted with NTFS, a Master File Table (MFT) file and other pieces of metadata are created. Metadata are the files NTFS uses to implement the file system structure. NTFS reserves the first 16 records of the MFT for metadata files.

NTFS creates a file record for each file and a directory record for each directory created on an NTFS volume. The MFT includes a separate file record for the MFT itself. These file and directory records are stored on the MFT. The attributes of the file are written to the allocated space in the MFT. Besides file attributes, each file record contains information about the position of the file record in the MFT.

Note that other file systems can and do deal differently with metadata.

EDIT: It has been pointed out in the comments section that this answer is missing the point because the OP asked for metadata on FAT32 filesystems, not NTFS. If I knew how, I would remove the "correct answer" attribute. Therefore I provide additional information that answers the question with regard to FAT32.

FAT32 saves simple metadata such as visibility or modification time for each file and folder in an entry in the parent folder of the the file or folder, creating a tree down from the root folder of the FS. As pointed out with regards to NTFS this is not a file but saved within the folder data structure. The entry originally was 32 byte large and contained the following attributes:

Name (8.3) xxxxxxxx.yyy. (88 bits)
Attribute byte (8 bits of information, described later in this section).
One reserved byte.
Create time (24 bits).
Create date (16 bits).
Last access date (16 bits).
Two reserved bytes.
Last modified time (16 bits).
Last modified date (16 bits).
Starting cluster number in the file allocation table (16 bits).
File size (32 bits).

The list was taken from this Microsoft Technet article and pertains to FAT16. Since the cluster size of FAT32 can be 32 bits and the name of files can be longer than 8.3 the table is not wholly accurate. To accomodate long file names and larger disks FAT32 modifies some behaviour which can be read up in the Wikipedia here but the basic idea holds.

Answer 2

Doesn't it occupy any size?!!

Yes, but it's a small entry in a large pre-allocated block. That block is counted in the "used" portion of your disk. Adding an entry inside that block doesn't require the block to be expanded.

Depending on the filesystem, eventually the block will be filled and extended somehow after a lot of filenames are added.

Can I see the metadata file in the flash memory?

Not easily

As Ruslan and Blorgbeard commented. You can install a hex viewer such as HxD which will allow you to view (and edit - be very careful) the raw filesystem data. But you'll have to do the interpretation yourself concerning which bytes belong to which filesystem structures. For that you would need some good documentation of the specific filesystem used on the flash disk. FAT32 is likely to be simpler to understand than any of the many variants of NTFS. See Understanding FAT32 Filesystems for example.

Answer 3

Metadata is not stored (nor reported by usual file management tools) as files, it is stored on filesystem's data on disk.

Depending on the nature / version of the filesystem, each entry will take some amount of disk space to represent the metadata information.

Moreover alongside the space allocated in the Master File Table, some filesystems will also keep journal about files change (taking extra space), and some filesystems can even be extended with extra fields containing special purpose metadata.

So technically metadata takes disk space, but it is not taken in account by most file management utilities which works calling system's API that query the filesystem for file space, not for metadata space.

And this abstraction is only the tip of the iceberg, as filesystem itself works on an abstraction of underlying physical disk space, provided by low level disk routines, so actually only the disk's internal logic will know how many actual memory positions are available for higher levels and how many are marked as not reliable, reserved, or used for checksum.

Answer 4

The metadata depends on the file system. The most basic file systems usually used on exchangeable media is based on a DOS file system (FAT). DOS doesn't have users, and permissions. Or, more correctly, some of that information is carried in the 8th data bit of the file name. The only resource overhead that is used in a basic DOS FS is to account for the blocks of the file, or the directory that it is in - and that block is probably already committed for handling ".", already. IOW, adding an empty file wouldn't add another bit of storage, but it would change some existing consumed and allocated bits.

When you use more advanced file systems, with journalling and users, you get metadata, and journal entries and possibly a forensic chain to recover previous file versions, etc. Then a small file can explode in used storage.

So, check the FS type. If it is FAT, then you probably don't have a user recorded in metadata on the media. Hence... no space used. :)

A good example of a FAT FS is an open source implementation - and you can see that the list of operations available does not include "get/set user" and "check access by user". No identity = no storage of that identity.

Answer 5

Where are metadata kept?

When we talk about metadata, there are two types of metadata.

The first type includes created date, last modified date, last accessed date. Depending on file system (i.e. NTFS / FAT / Ext3 etc...) there will be different "metadata" available, for example Windows owner and permission on NTFS.

The first type applies to all files, e.g. the .txt file in your example.

All file systems pre-allocate spaces to contain these metadata at the Master File Table (MFT for NTFS, some other names for other file system), which is not directly accessible by users. Since MFT is not counted as part of drive capacity, there is no "additional" storage space required anyway.

The second type of "metadata" is defined by the file type or application. For example, Office documents keeps "Author", "Subject" and other metadata; JPEG images keeps a set of EXIF data including "date picture taken", "camera model", "shutter speed"; while MP3 sound contains "album", "track #", "bitrate"...

The second type DOES take up additional space, because these "metadata" forms part of the file.

---

Different size in different drive

When your text file is on C:\ it takes up 4K. It becomes 1K in size when you put it on your flash drive H:\. This is because different "block size" for the different partitions.

Files are allocated spaces in block. Therefore, on a file system of block size 4K, 1 byte is allocated 4K while 4,097 bytes (4K + 1 byte) is allocated 8K.

Apparently your C: is formatted with 4K block size while H: is formatted with 1K block size, resulting in the difference.

Answer 6

As you see in the last picture above, the only space that the file occupy on the flash, is the space for its contents [characters].So where is metadata file ?

The "metadata file" is the directory that contains the file. That's basically what a directory is -- a collection of metadata describing the contents of the directory.

I mean, when I move the file to another computer, how it understand name of file, Owner of file, Date created and modified and ... ?

Doesn't it occupy any size?!!

Yes, in the directory. In most filesytems, the same file can have two different names if it's linked into two different directories.

Can I see the metadata file in the flash memory?

If your filesystem supports it, you can see it by looking at the size of the directory.