Speeding up find command result with folder exclusions

Question

I have the following folder structure:

Approvals_Share
    1 CUST APPROVAL
    2 LICENSE APPROVAL
    CANADA INSCRIBE
    z-remove

There are multiple subfolders and illustrator files “.ai” inside each folder

If I perform a find individually on each folder I get the results back in the amount of time specified below between square brackets:

1 CUST APPROVAL: [14m 19s]
2 LICENSE APPROVAL: [10m 17s]
CANADA INSCRIBE: [13s]
z-remove: [2s]

Here is a sample shell command that I use to lists the z-remove folder

cd '/Volumes/Box/Approvals_Share/z-remove/'
find .  -name '*.ai' -not -name '.*' > output.txt

This task takes only 2 seconds to complete, however if I run the same command on the parent folder Approvals_Share excluding all folders except z-remove the task takes 3m 27s. I was expecting it to be 2s or way closer. Here is the shell command I’m using:

cd '/Volumes/Box/Approvals_Share/'
find .  -name '*.ai' -not -name '.*' -not -path './2 LICENSE APPROVAL/*' -not -path './1 CUST APPROVAL/*' -not -path './CANADA INSCRIBE/*' > output.txt

Is there a way to speed up the result?

Reda Salih · Answer 1 · 2020-03-13T14:35:08.487

Using not -path requires traversing all the excluded directories with ignoring the output, reason why it's taking more time.

The solution here consists of using -prune, it forbids find to access to the children of every pattern within -path option:

cd /Volumes/Box/Approvals_Share/
find . -path './2 LICENSE APPROVAL' -prune -o -path './1 CUST APPROVAL' -prune -o -path './CANADA INSCRIBE' -prune -o -name '*.ai' -not -name '.*' -print

Another way consists of grouping the excluded directories :

find . -type d \( -path './2 LICENSE APPROVAL' -o -path './1 CUST APPROVAL' -o -path './CANADA INSCRIBE' \) -prune -o -name '*.ai' -not -name '.*' -print

score 0 · Answer 2 · answered Mar 11 '20 at 23:12

This other answer is right but it requires you to explicitly specify all directories you want to exclude. The solution doesn't scale up well. What if there are hundreds of directories in .?

Your requirement is to exclude all directories except z-remove (and . implicitly in the first place, otherwise it wouldn't make sense). A more scalable solution should specify z-remove (and .) only, as this information is enough to exclude the rest.

If you know there are no matching files directly in . then all of them (if any at all) will be in z-remove. In such case use this straightforward approach:

find z-remove -name '*.ai' ! -name '.*'

^{(Note: I used ! because it's POSIX while -not is not.)}

If there can be matching files directly in . then you need to start in . (obviously) and prune every directory that is not . or ./z-remove or inside ./z-remove. Like this:

find . -type d ! -path . ! -path ./z-remove ! -path './z-remove/*' -prune \
       -o -name '*.ai' ! -name '.*' -print

Note your original code does not use -type f, so e.g. a directory ./z-remove/foo.ai will pass all the tests. You're talking about files and in Unix directory is also a file. But you're talking about "illustrator files", so maybe you mean regular files and -type f is what you want. If so, my solution becomes:

find . -type d ! -path . ! -path ./z-remove ! -path './z-remove/*' -prune \
       -o -type f -name '*.ai' ! -name '.*' -print

Speeding up find command result with folder exclusions

2 Answers2