0

Files

sample$1.class
sample.class
scp_test.sh

scp_test.sh

#!/bin/bash

TARGET=('sample.class' 'sample$1.class')


for dd in "${TARGET[@]}"
do
  FILENAME=basename ${dd}
  scp ${FILENAME} remote:/tmp/${dd}
done

When the shell is run, sample$1.class is overwritten on sample.class on remote server,
leaving only sample.class.

$ ./scp_test.sh
sample.class               100%    0     0.0KB/s   00:00
sample$1.class             100%    0     0.0KB/s   00:00
$ ssh remote 'ls -l /tmp/'
total 01 
sample.class

It only happens when it's copied by scp. Local copy using cp doesn't behave like this.

*EDIT Since I use Bash version under 4.3(4.2.46), I can't use ${dd@Q}. I fixed the problem by using quote combiniation '"${dd}"' instead of plain ${dd}.

Lunartist
  • 159

1 Answers1

1

Preliminary note

This answer is mainly about traditional scp, i.e. legacy scp using the SCP protocol. Modern scp from OpenSSH uses SFTP by default and when I'm writing this the transition is less than two months old; so it's quite fresh. I can tell your scp uses SCP, because if it used SFTP, you wouldn't encounter the issue in question. Solving such issues was one of the reasons SCP was replaced by SFTP.

From the changelog:

OpenSSH 9.0 was released on 2022-04-08. […]

[…]

This release switches scp(1) from using the legacy scp/rcp protocol to using the SFTP protocol by default.

Legacy scp/rcp performs wildcard expansion of remote filenames (e.g. scp host:* .) through the remote shell. This has the side effect of requiring double quoting of shell meta-characters in file names included on scp(1) command-lines, otherwise they could be interpreted as shell commands on the remote side.

Potential issues independent from scp

First of all: quote right. Your example names may be safe even if unquoted, still writing robust code is a virtue. IMO it's easier to always quote than to think hard each time if you can get away without quoting. Your code with properly quoted variables will look like this (note I don't attempt to solve the issue with scp yet, this will be done in a moment):

#!/bin/bash

target=('sample.class' 'sample$1.class')


for dd in "${target[@]}"
do
  filename="$(basename "${dd}")"
  scp -- "${filename}" "remote:/tmp/${dd}"
done

(I also fixed names all in capitals, introduced double dash in case you ever add a filename starting with a dash to target. I think basename is a no-op in this particular case, but I assume you have a reason for it.)

If your scp used SFTP then the above code would work (and frankly I think your original code would also work with such scp, but only because the names you used are "safe" when unquoted).

The issue with scp

Unfortunately the legacy scp embeds the remote pathname in a shell code that is meant to be interpreted by a remote shell (compare this answer of mine). The remote shell will interpret characters like quotes, $, [ etc., unless they are escaped or quoted when they get to the remote shell. This means locally you need to additionally quote with the remote shell in mind. You need to quote for the local shell and for the remote shell. This is what the citation means by "requiring double quoting of shell meta-characters in file names".

Solution

Bash can help. "${dd@Q}" will expand dd and escape or quote the result, so after an additional level of interpretation (performed by the remote shell in your case) the result will be a single word you would expect (like "$dd" expanded locally). The following line is a fix for the legacy scp:

scp -- "${filename}" "remote:/tmp/${dd@Q}"

The whole script will be:

#!/bin/bash

target=('sample.class' 'sample$1.class')


for dd in "${target[@]}"
do
  filename="$(basename "${dd}")"
  scp -- "${filename}" "remote:/tmp/${dd@Q}"
done

The solution is not portable, it won't work in pure sh. The shebang in your script is #!/bin/bash from the beginning, so I guess you don't mind code that only works in Bash.

Final notes

  • The first snippet (without ${dd@Q}) is right for scp using SFTP (i.e. the new one). The second snippet (with ${dd@Q}) is right for scp using SCP (i.e. the legacy scp). There is no simple universal code. The new scp supports -O that makes it behave like the legacy scp, but if you use a legacy scp with -O then it will fail because it will find the option invalid. Either way you need to know if your scp is new or old, and adjust your shell code accordingly. For now your scp is old (and hence the issue in the first place), but if you update it, it may be replaced by a new one. The already linked changelog notices the incompatibility:

    This creates one area of potential incompatibility: scp(1) when using the SFTP protocol no longer requires this finicky and brittle quoting, and attempts to use it may cause transfers to fail. We consider the removal of the need for double-quoting shell characters in file names to be a benefit and do not intend to introduce bug-compatibility for legacy scp/rcp in scp(1) when using the SFTP protocol.

  • Side "issue", a friendly advice. Naming your scripts like scp_test.sh is not a good habit. For your scp_test.sh the interpreter is already bash, not sh. Your original code needs bash and doesn't work in pure sh (because of the array). Are you going to rename to scp_test.bash? What if you ported the script to Python? OK, I don't think you will do this to the very script in question, but in general you might. Then every tool that calls scp_test.sh would need to be fixed and call scp_test.py instead.

    Name the script scp_test. You can tell if scp_test is executable by examining its permissions. If you want to know what it is, invoke file scp_test. Now you can rewrite the script in whatever language you want, or even compile a binary, and you (or anyone/anything) can still run it as scp_test.

Kamil Maciorowski
  • 81,893