scp multiple arguments of bash script

Question

I am trying to use scp to copy multiple files from a remote server.

I can successfully do it for a single argument as script file_number

Where script is:

#!/bin/bash
echo Insert your username:
read user
scp $user@server:/path/run$1*.lz4 ./
lz4 -mdv --rm run$1*.lz4

All my files have the format: runXXXXsubY.lz4

This way I can copy all the files that have XXXX==file_number, but I want to pass multiple arguments, and copy all those files e.g. script 1 2 to copy all the files with xxxx==1 and XXXX==2.

I tried with:

scp $user@server:/path/run"$@"*.lz4 ./

But $@ gets expanded without appending the *.lz4

I appreciate any help in the right direction.

Thanks

It is fraught with peril and quoting issues, but you could do `scp $(printf "$user@server:/path/run%s.lz4 " "$@") ./`. You're better off with xargs. — William Pursell, Sep 25 '21 at 23:17
You're better off with a simple for loop calling `scp` once for each file. There's not much benefit from calling `scp` with multiple args over calling it multiple times. — William Pursell, Sep 25 '21 at 23:18
@WilliamPursell The problem with the `for` loop is that it asks for the password as many times as there are arguments. Which is something I want to avoid. — user17004502, Sep 25 '21 at 23:20
You should set up your ssh keys so that no password is required. — William Pursell, Sep 25 '21 at 23:21
If you need to copy a *lot* of files, I would look at how to run `ssh` in master mode so that each `scp` can reuse the same authenticated connection for each transfer, rather than having to reauthenticate for each file. — chepner, Sep 26 '21 at 00:40
In `zsh` (if you'll forgive the digression), you could write `scp "$user"@server:/path/run"${^@}*".lz4 ./`, which would expand the way you like. — chepner, Sep 26 '21 at 00:47

Gordon Davisson · Accepted Answer · 2021-09-26T01:30:39.537

The basic problem is that when the shell expands $user@server:/path/run"$@"*.lz4, it doesn't copy the $user:... and *.lz4 parts for each argument, it just kind of blindly adds the list of arguments -- including word breaks between arguments -- into the middle. So if the args are 1 and 2, it essentially expands to:

scp $user@server:/path/run"1" "2"*.lz4 ./

...so $user@server:/path/run"1" and "2"*.lz4 are separate arguments to scp, which isn't useful. What you can do is create an array based on the arguments, and then use that as the source list for scp. Something like this:

sources=()
for runNum in "$@"; do
    sources+=("$user@server:/path/run${runNum}*.lz4")
done
scp "${sources[@]}" ./

And then use a separate loop for the lz4 command:

for runNum in "$@"; do
    lz4 -mdv --rm run${runNum}*.lz4
done

EDIT: to avoid having to authenticate multiple times, you can open a master SSH connection and let all the scp transfers piggyback on that. Here's an example, based heavily on Felix Rabe's answer here:

# Create the array of source file patterns to fetch
sources=()
for runNum in "$@"; do
    sources+=("$user@server:/path/run${runNum}*.lz4")
done

# Open master SSH connection:
# (This is the only time you have to enter the password)
sshSocket=~/"$user@server"
ssh -M -f -N -S "$sshSocket" "$user@server"

# Actually copy the files:
scp -o ControlPath="$sshSocket" "${sources[@]}" ./

# Close master connection:
ssh -S "$sshSocket" -O exit "$user@server"

This solution still requires me to enter the password for every argument in the script. — user17004502, Sep 26 '21 at 00:16
@user17004502 I apparently didn't test that part thoroughly. In any case, it can be fixed with a master SSH connection, see update. — Gordon Davisson, Sep 26 '21 at 01:31

William Pursell · Answer 2 · 2021-09-25T23:54:30.020

Your best bet IMO is a simple for loop:

for xxx; do
    scp "$user@server:/path/run${xxx}.lz4" ./ 
done

If you do want to run a single scp (avoiding multiple password prompts is not a motivation for doing this; the way to avoid multiple password prompts is to setup your ssh keys so no password is necessary), you can use xargs:

printf "$user@server:/path/run%s.lz4\0" "$@" | xargs -0 -J % scp % ./

This does not guarantee a single invocation, and xargs will run multiple invocations of scp if enough args are passed that the argument string exceeds certain limits, but this should probably be sufficient.

markp-fuso · Answer 3 · 2021-09-26T00:32:45.933

0

One parameter substitution idea to build a pattern for the remote file names:

$ fn="$@"
$ echo scp "user@server:/path/run{${fn// /,}}*.lz4" ./

Assuming $@ = 222 333 this generates:

scp user@server:/path/run{222,333}*.lz4 ./

NOTE: once satisified with the format remove the echo

edited Sep 26 '21 at 00:32

answered Sep 25 '21 at 23:29

markp-fuso

28,790
4
16
36

scp multiple arguments of bash script

3 Answers3