Using Rsync include and exclude options to include directory and file by pattern

Question

I'm having problems getting my rsync syntax right and I'm wondering if my scenario can actually be handled with rsync. First, I've confirmed that rsync is working just fine between my local host and my remote host. Doing a straight sync on a directory is successful.

Here's what my filesystem looks like:

uploads/
  1260000000/
    file_11_00.jpg
    file_11_01.jpg
    file_12_00.jpg
  1270000000/
    file_11_00.jpg
    file_11_01.jpg
    file_12_00.jpg
  1280000000/
    file_11_00.jpg
    file_11_01.jpg
    file_12_00.jpg

What I want to do is run rsync only on files that begin with "file_11_" in the subdirectories and I want to be able to run just one rsync job to sync all of these files in the subdirectories.

Here's the command that I'm trying:

rsync -nrv --include="**/file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/

This results in 0 files being marked for transfer in my dry run. I've tried various other combinations of --include and --exclude statements, but either continued to get no results or got everything as if no include or exclude options were set.

Anyone have any idea how to do this?

ruakh · Accepted Answer · 2023-04-12T15:29:54.980

152

The problem is that --exclude="*" says to exclude (for example) the 1260000000/ directory, so rsync never examines the contents of that directory, so never notices that the directory contains files that would have been matched by your --include.

I think the closest thing to what you want is this:

rsync -nrv --include="*/" --include="file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/

(which will include all directories, and all files matching file_11*.jpg, but no other files), or maybe this:

rsync -nrv --include="/[0-9][0-9][0-9]0000000/" --include="file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/

(same concept, but much pickier about the directories it will include).

In either case, note that the --include=... option needs to come before the --exclude=... option, because we need the former to take precedence over the latter when a file matches both patterns.

edited Apr 12 '23 at 15:29

answered Mar 31 '12 at 01:02

ruakh

175,680
26
273
307

1

Thanks! That was exactly what I needed. My scenario was actually more or less what you described in your second example, but I kept my question simplified to make the question more straight forward. – Sheldon Chang Apr 01 '12 at 01:16
13

Note the importance of (e.g.) `--include="*/"` in including the **parent** directories of the files you actually want to include. – mabraham Jan 06 '16 at 14:13
29

Note the order of arguments: `--include` has to come before `--exclude` – hfs Apr 04 '16 at 07:59
@hfs Yeah, I feel that this should be mentioned in the actual answer. I was trying to do something similar before finding this page, and knew that I needed the `-include="*/"`, yet it still didn't work. Looking at this answer my first thought was "that's exactly what I'm doing!". Then I noticed the order was different. – Laurence Gonsalves Dec 06 '17 at 21:57
Another key concept is that "when using the --recursive (-r) option (which is implied by -a), every subcomponent of every path is visited from the top down, so include/exclude patterns get applied recursively to each subcomponent's full name" – wisbucky Nov 24 '20 at 04:09
For some odd reason this doesn't work in my case. I only want to copy some directories and all files in them with the subdirectories. I have the directory list in a file and use `--include-from=name-fo-file`. If I include the `--include="*/"` in the biginning, it copies everything. If I don't, it copies only the empty "included" directories. I'm puzzled. – Andyc Apr 12 '23 at 09:44

score 78 · Answer 2 · answered Dec 28 '17 at 16:03

rsync include exclude pattern examples:

"*"         means everything
"dir1"      transfers empty directory [dir1]
"dir*"      transfers empty directories like: "dir1", "dir2", "dir3", etc...
"file*"     transfers files whose names start with [file]
"dir**"     transfers every path that starts with [dir] like "dir1/file.txt", "dir2/bar/ffaa.html", etc...
"dir***"    same as above
"dir1/*"    does nothing
"dir1/**"   does nothing
"dir1/***"  transfers [dir1] directory and all its contents like "dir1/file.txt", "dir1/fooo.sh", "dir1/fold/baar.py", etc...

And final note is that simply dont rely on asterisks that are used in the beginning for evaluating paths; like "**dir" (its ok to use them for single folders or files but not paths) and note that more than two asterisks dont work for file names.

Your answer is the only one which is usable because you explain the general behaving. The other answers are too specific according the OP but each situation needs another solution! It helped me a lot! — Peter VARGA, Feb 04 '18 at 11:00
A thorough yet not as intuitive explanation can be found on the `rsync` manpage in section `INCLUDE/EXCLUDE PATTERN RULES` — Gerrit-K, Aug 02 '19 at 08:44

score 26 · Answer 3 · answered Nov 24 '20 at 08:18

Here's my "teach a person to fish" answer:

Rsync's syntax is definitely non-intuitive, but it is worth understanding.

First, use -vvv to see the debug info for rsync.

$ rsync -nr -vvv --include="**/file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/

[sender] hiding directory 1280000000 because of pattern *
[sender] hiding directory 1260000000 because of pattern *
[sender] hiding directory 1270000000 because of pattern *

The key concept here is that rsync applies the include/exclude patterns for each directory recursively. As soon as the first include/exclude is matched, the processing stops.

The first directory it evaluates is /Storage/uploads. Storage/uploads has 1280000000/, 1260000000/, 1270000000/ dirs/files. None of them match file_11*.jpg to include. All of them match * to exclude. So they are excluded, and rsync ends.

The solution is to include all dirs (*/) first. Then the first dir component will be 1260000000/, 1270000000/, 1280000000/ since they match */. The next dir component will be 1260000000/. In 1260000000/, file_11_00.jpg matches --include="file_11*.jpg", so it is included. And so forth.

$ rsync -nrv --include='*/' --include="file_11*.jpg" --exclude="*" /Storage/uploads/ /website/uploads/

./
1260000000/
1260000000/file_11_00.jpg
1260000000/file_11_01.jpg
1270000000/
1270000000/file_11_00.jpg
1270000000/file_11_01.jpg
1280000000/
1280000000/file_11_00.jpg
1280000000/file_11_01.jpg

https://download.samba.org/pub/rsync/rsync.1

Using Rsync include and exclude options to include directory and file by pattern

3 Answers3

Linked