command works in CLI but not in bash

Question

Doing a simple read in bash with this:

contents of list.txt:
/foo/bar/mydirectory/myfile.jpg
/foo/bar/mydirectory/deletedfile.jpg
/foo/bar/pictures\ of\ coffee\ cups/coffee-cup-42.jpg

#!/bin/bash

file="/foo/bar/list.txt"

while read -r line; do

    echo "VALUE OF LINE VARIABLE IS: $line"
    echo "COMMAND LINE IS: find -f $line"
    find -f $line
    # either file found, or "no such file" error
    done <$file

Output of script:

dumbjoe$ ./read-test.sh
VALUE OF LINE VARIABLE IS: /foo/bar/mydirectory/myfile.jpg
COMMAND LINE IS: find -f /foo/bar/mydirectory/myfile.jpg
/foo/bar/mydirectory/myfile.jpg file is found
VALUE OF LINE VARIABLE IS: /foo/bar/mydirectory/deletedfile.jpg
COMMAND LINE IS: find -f /foo/bar/mydirectory/deletedfile.jpg
find: /foo/bar/mydirectory/deletedfile.jpg: No such file or directory file not found
VALUE OF LINE VARIABLE IS: /foo/bar/pictures\ of\ coffee\ cups/coffee-cup-42.jpg
COMMAND LINE IS: find -f /foo/bar/pictures\ of\ coffee\ cups/coffee-cup-42.jpg
find: /foo/bar/pictures\: No such file or directory
find: of\: No such file or directory
find: coffee\: No such file or directory
find: cups/coffee-cup-42.jpg: No such file or directory WHAT???

running the command in Terminal
dumbjoe$ find -f /foo/bar/pictures\ of\ coffee\ cups/coffee-cup-42.jpg
/foo/bar/pictures\ of\ coffee\ cups/coffee-cup-42.jpg file found

Why is this not working in the script??

EDIT: Ultimeately what I'm after is below, where I'm getting lost is the variable "multi-escaping":

file=$LISTOFFILES

while IFS= read -r line

    do
    
    let "FILESTESTED+=1"
    
    FOUND="$(find -f $line)"
    
    # if file is not found, ignore the error
    exec 2> /dev/null
    
    # if file is found:
  
        if [ "$FOUND" == "$line" ]
            then
        
                echo "FOUND: $line" >> $REPORT
        
                FILESIZE="$(find $line -exec ls -l {} \; |  awk '{ print $5 }')"
    
                echo "SIZE is: $FILESIZE" >> $REPORT
                echo "-----------------------------------" >> $REPORT
    
                let "SPACETOTAL= SPACETOTAL + FILESIZE"
                let "FILES_FOUND_COUNT+=1"

        fi
    done <$file

All of this works IF there are no spaces present in the path.

See: [Why does shell ignore quoting characters in arguments passed to it through variables?](https://stackoverflow.com/questions/12136948/why-does-shell-ignore-quoting-characters-in-arguments-passed-to-it-through-varia) (it's about quotes, not escapes, but the same problem applies to both.) — Gordon Davisson, Sep 03 '21 at 02:44
@TomS : As you can see from the output you get, your path names contain spaces. Therefore the `$line` is split on those spaces. There are two workarounds: Either quote your variables, or replace `#!/bin/bash` by `#/bin/zsh`, because as a zsh-script, it would be correct. — user1934428, Sep 03 '21 at 06:43
@user1934428 As a zsh script, it would perhaps do what the OP expects, but its behavior would be in violation of the shell standard. I don't think "correct" is the appropriate description for such behavior. — William Pursell, Sep 03 '21 at 11:28
@WilliamPursell : Well, a _shell standard_ (whatever this means) would probably be a POSIX shell script, so the #! line should not be _bash_ either. I don't know to what extend the OP wants to achieve portability. From what i know, the platforms where you don't have zsh installed are rare, but in the end, this is something the OP needs to decide. — user1934428, Sep 03 '21 at 13:45

score 1 · Accepted Answer · answered Sep 02 '21 at 21:27

1

You don't need to escape the spaces in the input file, because the shell won't process them. Just quote the parameter expansion.

Put the following in list.txt:

/foo/bar/mydirectory/myfile.jpg
/foo/bar/mydirectory/deletedfile.jpg
/foo/bar/pictures of coffee/coffee-cup-42.jpg

Then change your script to

#!/bin/bash

file="/foo/bar/list.txt"

while IFS= read -r line; do

  echo "VALUE OF LINE VARIABLE IS: $line"
  echo "COMMAND LINE IS: find -f $line"
  find -f "$line"
  # either file found, or "no such file" error
done < "$file"

answered Sep 02 '21 at 21:27

chepner

497,756
71
530
681

That doesn't split the path, but it does NOT find the existing files: – Tom S Sep 03 '21 at 19:06
VALUE OF LINE VARIABLE IS: /foo/bar/pictures of coffee cups/white coffe cup.jpg COMMAND LINE IS: find -f /foo/bar/pictures of coffee cups/white coffe cup.jpg find: /foo/bar/pictures of coffee cups/white coffe cup.jpg: No such file or directory dumbjoe$ ls pictures\ of\ coffee\ cups/ coffee_cup_42.jpg white coffee cup.jpg dumbjoe$ – Tom S Sep 03 '21 at 19:08
sorry, idk how to format that bette within the comment – Tom S Sep 03 '21 at 19:09
I don't know if you have typos in your comments or the code you are running. At the very least, `coffee_cup_42.jpg` is not the same as `coffee-cup-242-jpg`. – chepner Sep 03 '21 at 19:19
errors crept in as I was "sanitizing" the post for brevity. No typos in code, list, or filenames. First thing I checked, along with actual spaces, special & invisible chars present etc... – Tom S Sep 03 '21 at 20:58

score 1 · Answer 2 · answered Sep 02 '21 at 21:28

The actual filepath is /foo/bar/pictures of coffee cups/coffee-cup-42.jpg (no backslashes).

This:

find -f /foo/bar/pictures\ of\ coffee\ cups/coffee-cup-42.jpg

is equivalent to this:

find -f '/foo/bar/pictures of coffee cups/coffee-cup-42.jpg'

which works. (The backslashes are just a quoting mechanism, similar to single– or double-quotes.)

By contrast, this:

variable='/foo/bar/pictures\ of\ coffee\ cups/coffee-cup-42.jpg'
find -f $variable

is equivalent to this:

variable='/foo/bar/pictures\ of\ coffee\ cups/coffee-cup-42.jpg'
find -f '/foo/bar/pictures\' 'of\' 'coffee\' 'cups/coffee-cup-42.jpg'

which sends too many separate arguments to find, and some of these arguments have spurious backslashes at the end; so the command doesn't work work.

The main issue is that backslashes are specifically a feature of how you type Bash commands. They work just as well in scripts as when you type at the shell, but they only work when they're actually in a command itself, not just in a variable that you pass into a command.

A secondary issue is that Bash has a weird feature called "word splitting" whereby, if your variable contains whitespace and you don't wrap the expansion in double-quotes, it finds the space and splits it up into separate arguments. This is a terrible feature IMHO, but it's specified by POSIX and I don't expect it to ever change. Fortunately, there's a simple workaround, namely: always wrap your variable expansions in double quotes.

So, to fix this:

Modify your file to remove the spurious backslashes that aren't part of the actual filepaths.
Write find -f "$line" instead of find -f $line.

Perhaps it is unfortunate that field splitting occurs after parameter expansion, but it's not really "weird". — William Pursell, Sep 02 '21 at 21:31
@WilliamPursell: Aside from word splitting and filename expansion, the result of a parameter expansion or command substitution is treated literally -- no special handling of single-quotes, double-quotes, backslashes, exclamation points, tildes, backticks, dollar signs, etc. -- which makes sense to me. It would be one thing if there were a special notation to expand a parameter and then perform word splitting and filename expansion on the result, but just `$line` doesn't strike me as the obvious notation for that. ;-) — ruakh, Sep 03 '21 at 02:40

score 0 · Answer 3 · answered Sep 02 '21 at 21:27

0

Looks like an escaping issue maybe try find -f '$line'

also change your #! to #!/bin/bash -x and you'll get a bunch of helpful debugging info.

answered Sep 02 '21 at 21:27

WillStaves

11
2

command works in CLI but not in bash

3 Answers3