SCP off the most recently modified file in a remote directory

Question

Is there a simple command to SCP the most recently modified file in a directory on a remote host? I can figure out how to do that from local to remote... something like: scp ``ls -Art | tail -n 1\`` usr@remote:/var/log/yeet How can I do the same thing but from remote to local. (So get the most recently modified file from yeet and copy it to the local host)

Answer 1

There are few issues here.

Parsing ls

First of all you shouldn't parse ls. Your ls -Art | tail -n 1 is flawed and it cannot be reliably fixed. A standard reliable replacement is find and/or working with null-terminated lines.

Additionally I assume you need the most recently modified file (not a directory) directly in the current directory (not in a subdirectory).

The ability to parse input items in a form of null-terminated list is not usually required by POSIX. If your tools are rich enough with options, this is a robust replacement for your ls -Art | tail -n 1:

find . -maxdepth 1 -type f -printf '%T@ %p\0' |
   sort -zrn |
   head -zn 1 |
   cut -z -d " " -f 2-

It yields the result as a null-terminated filename. To do something with it you need to pipe it to xargs -0 …, e.g.:

… | xargs -0r cp -t "/target/dir/" --

or (if your cp doesn't support -t):

… | xargs -0r -I {} cp -- {} "/target/dir/"

In these commands there are a lot of things not required by POSIX (thus non-portable), including -- which makes cp stop parsing for options, so e.g. a file -R won't trigger recursion instead of being copied. Note that filenames leaving our find . … begin with . for sure, so -- can safely be omitted. I used it just to point out a good general practice when dealing with cp. Another good practice is quoting paths: the literal /target/dir/ would work without the quotes, but after you replace this example with your specific target path, you may need quotes, so I'm using them anyway.

In your (local to remote) case you would use scp instead of cp. It may have its own quirks while parsing arguments.

A POSIX-compliant but poorly performing command may be:

find . ! -name . -prune -type f -exec sh -c '
   [ "$(find . ! -name . -prune -type f -newer "$1" -exec printf a \; |
   wc -c)" -eq 0 ]' sh {} \; -print

It adapts an approach from this another answer to substitute (non-POSIX) -maxdepth 1. It spawns sh to reliably nest two find … -exec … clauses. The outer find probes all files and supplies them one by one to the inner find. The inner find finds all files newer than the given file, printing a single character (a) for every newer file. wc -c counts these characters. If there are none, it means the given file is the most recently modified; only then the outer find prints it.

There are few scenarios when the outer find may print more than one file:

• there are two or more files with the same "newest" mtime;
• files in the directory are modified while the command is running;
• the inner find cannot stat some files.

For this reason -quit as the final action of the outer find would be useful (note it would be useful with the inner find too, but for a different reason). Unfortunately -quit is not POSIX.

I used -print (-print0 is not POSIX), still non-standard filenames are not a problem because you don't need to pipe the output to another command. Just use -exec that deals with all possible filenames right; e.g. instead of -print you use:

-exec yet_another_command {} \;

Now you know how to find the most recently modified file in a local directory without parsing ls.

Finding files on a remote system

Whatever approach you choose (including the flawed ls … | tail …), you need to run the command on the remote system (or not, I will get to this exception later) in order to find the desired file in a remote directory.

The most obvious approach is to ssh into the remote system. In the new context the remote system is local and your local computer is remote. (Here I'm using the above POSIX-compliant command as the example, but you can use our first find … | sort -z … | head -z … | cut -z … | xargs -0 … if only the remote system supports all the required options).

ssh usr@remote
# now on the remote system
cd "/source/dir/" &&
find . ! -name . -prune -type f -exec sh -c '
   [ "$(find . ! -name . -prune -type f -newer "$1" -exec printf a \; |
   wc -c)" -eq 0 ]' sh {} \; -exec scp {} usr@local:"/target/dir/" \;

Note if you'd like to avoid cd and use find /source/dir … then there are more .-s that need to be replaced by /source/dir. It's way easier with cd.

You need your local system to be available via SSH from the remote one. If there's NAT on the way, you can bypass it with a remote port forwarding, something like this:

ssh -R 12322:127.0.0.1:22 usr@remote
# now on the remote system the same cd + find as above
# only the scp part is different 
… -exec scp -P 12322 {} usr@127.0.0.1:"/target/dir/" \;

Note it makes the remote port 12322 lead to your local sshd and any user of the remote system may try to misuse it. Another problem: the remote system may be configured to disallow you to forward a port in the first place.

You may want a single command to invoke on the local system. In this case a proper quoting is required and it gets even more cumbersome:

ssh usr@remote '
   cd "/source/dir/" &&
   find . ! -name . -prune -type f -exec sh -c '"'"'
      [ "$(find . ! -name . -prune -type f -newer "$1" -exec printf a \; |
      wc -c)" -eq 0 ]'"'"' sh {} \; -exec scp {} usr@local:"/target/dir/" \;
   '

I expect trouble if the remote scp needs to ask for your password though. For this reason or if you cannot run/reach your local sshd, you may need yet another approach.

This local command will print the desired filename obtained from the remote system:

ssh usr@remote '
   cd "/source/dir/" &&
   find . ! -name . -prune -type f -exec sh -c '"'"'
      [ "$(find . ! -name . -prune -type f -newer "$1" -exec printf a \; |
      wc -c)" -eq 0 ]'"'"' sh {} \; -print
   '

You can use it with local tools like xargs and scp. Note that parsing what -print yields is only slightly better than parsing ls. Use -print0 (non-POSIX, may not be available) or -exec printf "%s\0" {} \; (should work) instead of -print to obtain the desired filename as a null-terminated string. Now it's up to you what you'll do with it on the local side. This is useful if you need to have a POSIX-compliant remote command but tools in your local system are rich with options.

Notable exception: sshfs

sshfs allows you to mount usr@remote:"/source/dir/" as a /local/path/. See this answer of mine, I won't repeat myself to cover all the details. In your case the (local) procedure with rich (not limited to POSIX) tools is like:

sshfs usr@remote:"/source/dir/" "/local/path/"
find "/local/path/" -maxdepth 1 -type f -printf '%T@ %p\0' |
   sort -zrn |
   head -zn 1 |
   cut -z -d " " -f 2- |
   xargs -0r cp -t "/target/dir/" --
fusermount -u "/local/path/"

This is great. Everything you do you do with local tools. If only you can use sshfs the tools on the remote side and their available options no longer matter. Nor does it matter if you can reach your local system from the outside. And the method is the same: remote to local or local to remote, it doesn't matter.