copy|move sparse files on NTFS with Windows

Question

How can I copy or move a sparse file from one NTFS volume to another NTFS volume while maintaining the sparseness using the native tools included with Windows? If there is no way to do this with the native tools, then what is a free application from a trusted vendor that will do this? Surely there is a command prompt command or powershell command that will do this.

I have a file that represents 250GiB that is taking up 20GiB of disk space and I would like it to continue only taking up 20GiB when I move it. Thanks.

Answer 1

You cannot do it with standard Windows tools.

But there is an utility. Go to http://www.flexhex.com/docs/articles/sparse-files.phtml, download and unpack sparse.zip.

Then run from from the command line:

cs.exe c:\src.dat d:\dest.dat

The file will be copied preserving sparseness.

Answer 2

Use Cygwin and mount the filesystem with the sparse option. Then Unix commands that support sparse files, such as cp, dd conv=sparse, and rsync -S, will correctly create or copy the file as a sparse one. As one would expect, simple output redirection will not create a sparse file.

See the following demonstration.

$ mount -o sparse D: /tmp/mnt
mount: warning - /tmp/mnt does not exist.
$ cd /tmp/mnt
$ dd conv=sparse if=/dev/zero seek=10000 of=sparse count=1
1+0 records in
1+0 records out
512 bytes copied, 0.0101909 s, 50.2 kB/s
$ ls -lh sparse
-rw-rw-r--+ 1 dds None 4.9M Sep  5 13:05 sparse
$ du -h sparse
0       sparse
$ cp sparse sparse-cp
$ dd conv=sparse if=sparse of=sparse-dd
10001+0 records in
10001+0 records out
5120512 bytes (5.1 MB, 4.9 MiB) copied, 0.0500354 s, 102 MB/s
$ rsync -S sparse sparse-rsync
$ cat sparse >sparse-fail-cat
$ cat sparse | dd conv=sparse of=sparse-cat-dd
$ ls -lh sparse*
-rw-rw-r--+ 1 dds None 4.9M Sep  5 13:05 sparse
-rw-rw-r--+ 1 dds None 4.9M Sep  5 13:15 sparse-cat-dd
-rw-rw-r--+ 1 dds None 4.9M Sep  5 13:06 sparse-cp
-rw-rw-r--+ 1 dds None 4.9M Sep  5 13:11 sparse-dd
-rw-rw-r--+ 1 dds None 4.9M Sep  5 13:19 sparse-fail-cat
----rw----+ 1 dds None 4.9M Sep  5 13:06 sparse-rsync
$ du -h sparse*
0       sparse
0       sparse-cat-dd
0       sparse-cp
0       sparse-dd
4.9M    sparse-fail-cat
0       sparse-rsync

Answer 3

I wrote a little script copy-sparse-windows.sh to copy a file, making it as sparse as possible. This needs msys2 or similar, specifically sh, truncate, and rsync.

It uses the windows fsutil tool to set the sparse flag, truncate to make a large empty file, and rsync to copy the non-zero content.

As I understand, some of the other answers do not create new sparse regions. I wanted to reduce the size of a large VM disk image, by adding sparse areas everywhere possible.

#!/bin/sh -euv
from=$1 to=$2
touch "$to"
fsutil sparse setflag "$to"
truncate --reference "$from" "$to"
rsync --archive --inplace --no-whole-file --verbose --progress "$from" "$to"

Answer 4

On the Windows side I ended up using WinHex to do this or rather its forensic brother X-Ways Forensics. The GUI has an option in the menus to copy files sparsely.