2

I'm using rsync to copy all files from one 3TB external disk formatted as HFS+ to another 3TB external disk formatted as exFAT.

I've double checked both disks are exactly the same size and formatted the destination disk right before I began the copy, and in fact the source disk has 38GB space, so there's about 2962 GB of files (1TB = 1000GB on my system).

rsync has failed around 90% of the way through having filled up the destination disk. Can this be explained by differences between HFS+ and exFAT? Is exFAT much more wasteful of disk space when storing files compared to HFS+?

phuclv
  • 30,396
  • 15
  • 136
  • 260
Tim MB
  • 125

2 Answers2

2

TL;DR

The reason is due to the big block size by default in exFAT. Just reformat it with a smaller block size and it'll work

However also note that macOS will create a lot of hidden ._ files when copying using Finder to store the file metadata, which consumes even more space compared to HFS+. You should copy using command line to avoid creating those files

Actually exFAT so simple that it's much more efficient in disk usage than other more advanced file systems like NTFS, ext4, APFS, HFS+... providing block size is the same, because the more complex a file system is, the more metadata it requires. Just format a new partition in any of those file systems and you'll see that the used space in the blank drive is much larger than exFAT:

However the thing is that exFAT has much bigger block size by default. A 3 TB drive will have 1 MiB block size compared to 4 KiB on other file systems. See 1-byte file takes up 2 MB (yes, megabytes) on 6 TB exFAT Veracrypt volume. Why? for the default cluster sizes on exFAT:

Volume size   Cluster size
  500 GiB          128 KiB
 1000 GiB          256 KiB
 1500 GiB          512 KiB
 2000 GiB          512 KiB
 3000 GiB            1 MiB
 4000 GiB            1 MiB
 5000 GiB            2 MiB
 8000 GiB            2 MiB
10000 GiB            4 MiB
20000 GiB            8 MiB

That means if you have a 640 KiB file, you waste nothing in HFS+ but 384 KiB when storing it in exFAT. As a result an exFAT disk will fill up much faster by default. You must reformat the exFAT partition and choose a smaller block size to waste less space

See also

phuclv
  • 30,396
  • 15
  • 136
  • 260
1

> much more wasteful of disk space - there will always be differences in efficiency between filesystems. And this will inevitably show as you get closer to maximum capacity of similarly sized disks.

Just a difference on allocation block size will make a huge impact, e.g. lesser efficiency when you have a lot of small files and a comparatively large allocation block size.

Hannu
  • 10,568