How do I remove all holes in a sparse file on Linux? On Windows, I believe the procedure is
fsutil sparse setflag <file> 0
What is the Linux equivalent?
(I know cp --sparse=never <file> <file>.cp; rm <file>; mv <file>.cp <file>; would get the job done, if free disk space permits, but I guess Linux has a in-place way of doing it? Like Windows?)
fallocate -l <length> <file> might achieve this; while it doesn't tell the filesystem to explicitly zero-fill the holes, it does allocate space for them. Compare with:
[make a sparse file]
$ echo yay | dd bs=1 seek=1G of=test.file
[list extents]
$ filefrag -v test.file
ext: logical_offset: physical_offset: length: expected: flags:
0: 262144.. 262144: 902523324.. 902523324: 1: 262144: last,eof
[allocate space]
$ fallocate -l $(stat -c %s test.file) test.file
[list extents again]
$ filefrag -v test.file
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 196607: 2219092376..2219288983: 196608: unwritten
1: 196608.. 262143: 2219301264..2219366799: 65536: 2219288984: unwritten
2: 262144.. 262144: 902523324.. 902523324: 1: 2219366800: last,eof
Use filefrag -v <file> or xfs_io -r -c "fiemap -v" <file> to list a file's extents. In this example, after fallocate there are no more holes as such; they're now covered by 'unwritten' extents that the filesystem will continue to immediately return zeros on read, but which now do have specific physical blocks assigned to them.
It would be technically possible to add an option to fallocate which uses FIEMAP to list holes and/or "unwritten" extents and zero-fill them, but unfortunately it doesn't have one at the moment and requires the zero-fill offset/length to be specified by hand.
If the file happens to get truncated in between the stat -c %s and the fallocate, the latter operation will grow it again to its previous size.
You might be able to avoid this using -n/--keep-size (which only pre-allocates space but doesn't grow the file) if you expect it to grow again, though the exact Linux semantics of allocating space beyond EOF are a bit unclear to me. (I suppose it's similar to the Windows 'fsutil file setEOF/setValidData'.)
On the other hand, if the file grew in between the two operations, it will not be truncated; the newly added data (or holes) will just remain unaffected by the fallocate. (That is, you will not lose any data; worst case you'll just need to do it again to get rid of newly added holes.)
Be careful with cp – in recent GNU Coreutils versions, it will automatically try to preserve sparseness using SEEK_HOLE. You might need to use cp --sparse=never to prevent this.
In addition, as of Coreutils 9.x, cp foo bar and even cat foo > bar will automatically use the FICLONE or copy_file_range() operations to clone files without actually copying their data if the filesystem supports this, e.g. on XFS or Btrfs or ZFS (similar to the "block cloning" feature in ReFS).
If this happens, the "copy" will exactly retain the original sparseness of the file, and you will see the shared flag for each extent in filefrag's output. You can use cp --reflink=never to prevent this.
(There is a flag for the fallocate system call to unshare extents, but it hasn't yet been exposed through the fallocate command-line tool, although you can do it through xfs_io which has the 'funshare' subcommand.)
