Linux: how to move a large file tree, deleting files during the copy (not after)

Question

I'd like to encrypt a disk in-place, from clear to ecryptfs (not boot disk, just disk with lots of data files).

I've created the following structure:

  /mnt/
   clear/ # the source files
   cipher/ # the ecryptfs ciphertext directory
   mounted.clear/ # the mounted ecryptfs directory

I'd like to do something like: mv /mnt/clear/* /mnt/mounted.clear/

The problem is that since this is a cross-filesystem move, mv will first copy then unlink each argument (see here). I don't have enough free space on this disk to duplicate some of the subdirectories under /mnt/clear.

I wish there was an mv --delete-during argument (or similar), but there isn't (not in GNU coreutils 8.3 anyway :). Any other easy ideas? I could write a quick script to do it, but would rather not if there's an easier solution/utility I can use.

score 2 · Accepted Answer · edited Jun 12 '20 at 13:48

This option of rsync:

--remove-source-files
This tells rsync to remove from the sending side the files (meaning non-directories) that are a part of the transfer and have been successfully duplicated on the receiving side.

rsync -av --remove-source-files /mnt/clear/ /mnt/mounted.clear

My tests indicate the tool removes files on the fly, not at the end.

Notes:

The slash (/) after /mnt/clear is important.
"Successfully duplicated" means you should lose nothing, even if there are problems.
You will need to remove directories in a separate step. Use find /mnt/clear/ ! -type d to make sure there are no stale non-directories.

If you cannot use rsync, consider this option of tar:

--remove-files
Remove files from disk after adding them to the archive.

cd /mnt/mounted.clear/ &&
( cd /mnt/clear/ && tar -vc --remove-files * ) | tar -x

My tests indicate the first tar removes files on the fly, not at the end. There are pitfalls though:

You would want the first tar to postpone removing a file until it is safely written in the destination directory. No way. The tool cannot know what happens in the remaining part of the pipe. As soon as it processes a file, it removes it. It may happen the second tar lags behind, data accumulates in the pipe buffer, the first tar eagerly removes files. In such case if the second tar fails for whatever reason then the pipe will break and the buffered content (from some already removed files) will be lost. Reducing the buffer (e.g. with stdbuf) may minimize damage but I'm not sure if you can make the solution totally safe.

^{I deliberately didn't use cd … && tar … | ( cd … && tar … ). In this version a possible failure of the second cd may occur after the first tar processes and removes something. In the original code both cd commands must succeed before the first tar runs.}
* doesn't match files starting with ..

Additional note:

In my tests files disappeared from directory listings with delay, i.e. few moments (sometimes a minute) after rsync or tar processed them. I use Btrfs and I suspect this is its quirk.

Linux: how to move a large file tree, deleting files during the copy (not after)

1 Answers1