0

This is a strange question. I want to copy some large files from one harddrive to another on the same machine (on Linux from the command line).

The problem is that makes everything very slow.

I don't care though, to make the copying much slower and less HDD intensive. Let the machine take it time with it...

Is there a way to make the copy slower, but the resources on the machine more available?

jerrro
  • 1

3 Answers3

2

To run a command with lower scheduling priority, using less resources, you could use the nice command:

The syntax is:

nice [OPTION] [COMMAND [ARG]...]

where niceness ranges from -20 (most favorable scheduling) to 19 (least favorable) using the following parameter:

-n, --adjustment=N
    add integer N to the niceness (default 10)

Instead of launching the program with the default priority, you can use the nice command to launch the process with a specific priority. For example, test.pl is launched with a "nice" value of 10 instead of the default 0 (this is a hyphen and not minus):

$ nice -10 perl test.pl

This will make the perl process get less scheduling slices from the kernel, thus getting less resources and leaving more for other processes.

For more information see A brief guide to priority and nice values in the linux ecosystem.

As regarding I/O scheduling, nice also has influence. Wikipedia Completely fair queueing makes this remark:

using nice(1) also modifies I/O priorities somewhat

For more information, see the article Linux I/O Schedulers.

You could also directly set the I/O priority of a process by using ionice(1):

ionice - set or get process I/O scheduling class and priority

The I/O scheduling and priority can be drastically modified, up to the lowest:

Idle
    A program running with idle I/O priority will only get disk
    time when no other program has asked for disk I/O for a
    defined grace period. The impact of an idle I/O process on
    normal system activity should be zero. This scheduling class
    does not take a priority argument. Presently, this scheduling
    class is permitted for an ordinary user (since kernel
    2.6.25).
harrymc
  • 498,455
1

A disk drive is interrupt driven and will just work to keep up with the data.

So the answer to your question is that you cannot slow down the copy.

If you truly need more resources when copying, then replace both source and target drives with fast SSD drives. That will solve your issue.

I copy 200 GB of machines from one computer to the other monthly or quarterly. Both machines have fast SSD drives.

There is no (practical) slowdown on either machine during the (near) hour copy.

I had the resource issue you mention with slow hard drives (XP days) and the solution then was to do the large copy when the machines were not needed otherwise.

0

I don't care though, to make the copying much slower and less HDD intensive. Let the machine take it time with it...

A manual solution is to limit the transfer with pv. For a single file a basic command can be like:

pv -L 10M /source/file > /destination/file

Note this simple command won't preserve attributes (like cp -p or GNU cp -a would) and cannot copy file hierarchies. Therefore for anything more demanding use tar. Use the fact you can pipe tar to tar like this:

(cd /source && tar -c dirA/ dirB/ file1 file2) | (cd /destination && tar -x)

and it's like cp -R. It's useful for copying to multiple destinations or via SSH, netcat, any pipe. To solve your problem we place pv between tars:

(cd /source && tar -c dirA/ dirB/ file1 file2) | pv -L 10M | (cd /destination && tar -x)

You can use tar -c * (but keep in mind * matches non-dot-files); you can use some advanced options of tar. Don't use compression, you want to throttle the uncompressed stream.

Now the best part: with pv -R (invoked aside) you can change the settings of a running pv. This way you can adjust the throttle anytime; you just need to know the PID of your limiting pv. Use pidof pv to find the PID; or maybe modify the main command in the first place, so the middle part is:

… | pv -P /dev/tty -L 10M | …

and pv itself will print its PID to the terminal. See man 1 pv for details.

Suppose the PID is 2276. To change the throttle run pv -R 2276 -L 100M (or 100K, or 1G). Tweak it, find the setting that's best for you. Change anytime.

This answer can be combined with the nice+ionice idea. I believe renice (not nice) and ionice -p will allow you to renice already running processes.

Kamil Maciorowski
  • 81,893