3

I have a very large sparse file on an EXT4 partition, and I want to copy that file to an NTFS partition. The file has a size of about 2 TB, but since it is a sparse file it uses only 700 GB on disk.

The NTFS partition where I want to copy that file to only has about 1.3 TB free space. So if I can copy the sparse file there, it should fit comfortably; but as non-sparse file this won't fit.

I tried to copy the file with a simple cp -a command, but it failed in the end with "no space left on device". Interestingly it failed while about 570 GB were still reported as free.

By checking the file size and its reported disk usage I can see that the file on NTFS partition is not sparse.

I also tried to create a sparse file manually, with either of these commands:

  • dd if=/dev/zero of=test1.bin bs=1 count=1 seek=100000
  • truncate -s 100000 test2.bin

On an EXT4 partition both of the created files were sparse (du -h shows that they use 0 bytes on disk). But on the NTFS partition each file actually uses 100 KB of disk space, according to du -h and also according to ls -lsk.

So: why can't I create sparse files on the NTFS partition? And what can I do to copy the large file as sparse file to the NTFS partition?

The system where this happens:

  • Debian 11 (Bullseye) x86-64
  • kernel package 5.10.92-1
  • ntfs-3g package 1:2017.3.23AR.3-4+deb11u1

According to mount, the NTFS partition is mounted like this:
/dev/sdg1 on /mnt/loop2 type fuseblk (rw,relatime,user_id=0,group_id=0,allow_other,blksize=4096)

My understanding is that the partition is mounted using the ntfs-3g driver - is that correct? And according to man ntfs-3g that driver supports sparse files. I also did not find any option to enable or disable that feature.

I did not create the NTFS partition myself, but used the existing partition that was on the drive when I bought it.

oliver
  • 520
  • 1
  • 4
  • 17

2 Answers2

4

cp --sparse=always should do the trick,

root@devad22:/t# truncate -s 1G ntfs
root@devad22:/t# mkfs.ntfs ntfs --force
ntfs is not a block device.
mkntfs forced anyway.
The sector size was not specified for ntfs and it could not be obtained automatically.  It has been set to 512 bytes.
The partition start sector was not specified for ntfs and it could not be obtained automatically.  It has been set to 0.
The number of sectors per track was not specified for ntfs and it could not be obtained automatically.  It has been set to 0.
The number of heads was not specified for ntfs and it could not be obtained automatically.  It has been set to 0.
Cluster size has been automatically set to 4096 bytes.
To boot from a device, Windows needs the 'partition start sector', the 'sectors per track' and the 'number of heads' to be set.
Windows will not be able to boot from this device.
Initializing device with zeroes: 100% - Done.
Creating NTFS volume structures.
mkntfs completed successfully. Have a nice day.
root@devad22:/t# mkdir mnt
root@devad22:/t# mount ntfs mnt
root@devad22:/t# truncate -s 10G 10G_sparse_file.ext
root@devad22:/t# cp --sparse=always 10G_sparse_file.ext mnt/should_be_10G_sparse.ext
root@devad22:/t# cd mnt
should_be_10G_sparse.ext
root@devad22:/t/mnt# du -h .
4.0K    .
root@devad22:/t/mnt# du -h --apparent-size .
11G .
root@devad22:/t/mnt# df -h -
df: -: No such file or directory
root@devad22:/t/mnt# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop2      1.0G  5.7M 1019M   1% /t/mnt
root@devad22:/t/mnt# 
root@devad22:/t# dd if=/dev/zero of=10GB_not_sparse.ext bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 14.0689 s, 763 MB/s
root@devad22:/t# cp --sparse=always 10GB_not_sparse.ext mnt/10GB_sparse_from_not_sparse.ext
root@devad22:/t# cd mnt
10GB_sparse_from_not_sparse.ext  should_be_10G_sparse.ext
root@devad22:/t/mnt# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop2      1.0G  5.7M 1019M   1% /t/mnt
root@devad22:/t/mnt# du -h .
4.0K    .
root@devad22:/t/mnt# du -h --apparent-size .
21G .
  • copying with cp --sparse=always to NTFS works great for me, both from sparse and from non-sparse, the result becomes sparse either way :) using Ubuntu 22.04 with ntfs-3g 2021.8.22
hanshenrik
  • 1,925
  • 3
  • 25
  • 38
2

Normally you can’t in Windows copy or create a sparse file if its nominal size exceeds the amount of available free space. In addition, Windows files may need to be flagged as sparse by the fsutil sparse command, or they are not sparse.

I suggest doing the copy from inside Windows, perhaps using a utility such as sparse.zip, found at the end of the article NTFS Sparse Files For Programmers.

The syntax of this command is simple:

cs.exe from-path to-path

If this can't be done on Windows because the file is only on Linux, try to tar the file on Linux, then untar it on the target Windows system using Windows Subsystem for Linux (WSL).

harrymc
  • 498,455