Linux script variables to SCP and delete files

Question

I am looking to set up a script to do the following:

1st: SCP a directory on the first day of month to another server

2nd: Delete the directory after successful transfer

The directory I need to move will always have a different name, and the lowest numbered one is always the one that needs to move:

2018/files/02/

2018/files/03/

So what im looking to write up is something like:

scp /2018/files/% user@host:/backups/2018/files/
{where % = lowest num} && 
rm -rf /2018/files/%
{where % = lowest num} &&
exit

Thanks for any advice

Are you open to using the Ruby language? Ruby (or Python, or other scripting languages) would be easier to work with for something like this. — Keith Bennett, Feb 17 '18 at 11:30
@KeithBennett +1 It can be done in compact pure shell for the essential question of finding the directory. Check my answer below. It *does* depend on very specific shell behaviour, which must be understood well. — Henk Langeveld, Feb 17 '18 at 11:38
@HenkLangeveld I have no doubt that this can be done in pure shell (and I salute you for doing it!), but for me I am much more productive with nontrivial scripts in Ruby. Ruby is best known for web development in Rails but I find it's a super language for general purpose stuff. — Keith Bennett, Feb 17 '18 at 11:59

Keith Bennett · Answer 1 · 2018-02-17T12:14:37.110

If you are open to using Ruby, you could accomplish it with something like this:

def file_number(filespec)
  filespect.split('/').last.to_i
end

directories = Dir['/2018/files'].select { |f| File.directory?(f) }
sorted_dirs = directories.sort_by do |dir1, dir2|
  file_number(dir1) <=> file_number(dir1)
end

dir_to_copy = sorted_dirs.first
destination_dir = File.join('/', 'backups', dir_to_copy)

`scp #{dir_to_copy} user@host:#{destination_dir}`
`rm -rf #{dir_to_copy}`

I have not tested this, but if you have any problems, let me know what they are and I can work through it with you.

While using shell scripting eliminates the need for the Ruby interpreter, to me the code is not nearly as straightforward.

In very large directory lists (maybe 10,000's?) the sort might be intolerably slow, and another method would be needed to optimize for speed.

I would caution you against doing an unconditional rm -rf after the backup -- that seems really risky to me.

Henk Langeveld · Answer 2 · 2018-02-17T11:59:45.970

The big challenge here is to actually find the right files to copy, and shudder, delete. So let us call that step 0.

Let's start with some boiler plate

sourceD=/2018/files/ 
targetD=/backups/2018/files/

And a little assertion, which bails out from the script if $1 does not equate to a directory.

assert_directory() { (cd ${1:?directory name}) || exit; }

step 0: Identify directory:

assert_directory $sourceD
to_be_archived=$(
  # source must be two characters, hence "??"
  # source must a directory, hence trailing "/"
  # set -- sorts its arguments
  # First match must be our source
  set -- $sourceD/??/ &&
    assert_directory "$1"
    echo ${1:?nothing found}
) || exit

This is only a couple of lines of condensed code. Note that this may cause trouble if you (accidentally) run this multiple times in a row.

Step 1, Copy files now appears to be the easy part.

scp -r ${to_be_archived:?} user@host:${targetD:?}

This is a simple method for copying files, but also slow and risky. Lookup rsync over ssh for alternatives.

Step 2, Remove

The rm -fr line will do the job, but I won't include that here. We are missing an essential step, as we need to make sure that our files have arrived safely. Again, rsync has options for that.

In summary:

assert_directory() { (cd ${1:?directory name}) || exit; }

assert_directory $sourceD
to_be_archived=$(
  set -- $sourceD/??/ &&
    assert_directory "$1"
    echo ${1:?nothing found}
) || exit

This will give you the first two-character name directory (if one exists) in sourceD or abort the running script. It will break if $sourceD contains spaces.

Linux script variables to SCP and delete files

2 Answers2