5

I want to recursively loop through each file in a folder from bash and do some sort of SIMPLE manipulation to them. For example, change permission, change timestamp, resize image with ImageMagick, etc., you get the idea.

I know (like most beginners) on how to do it in a directory, but recursively... ?

$ for f in *.jpg ; do convert $f -scale 25% resized/$f ; done

Let's just keep it simple. say,

$ for f in *; do touch $f; done
wjandrea
  • 28,235
  • 9
  • 60
  • 81
GaurangiS
  • 85
  • 1
  • 6
  • So, show what you've got so far. - and also, "an assignment" is usually not just _one_ problem. We don't know what you need help with so please be specific. – Ted Lyngmo Jul 04 '20 at 03:04
  • `$ for f in *.jpg ; do convert $f -scale 25% resized/$f ; done` – GaurangiS Jul 04 '20 at 03:08
  • 1
    @GaurangiS - your loop isn't recursive. The simplest solution is to write a short script that takes a single filename as an argument and performs all your needed manipulations on it. Call is say `fixfiles` and make it executable `chmod +x fixfiles`. Now just using `find` with the `-exec` option and call your `fixfiles` script on each file, e.g. with `fixfiles` in `$HOME`, you can do `find . -type f -exec $HOME/fixfiles '{}' \;` to operate on every file in the current directory and below applying `fixfiles` to them. (think of `fixfiles` as a helper-script called by `find`) – David C. Rankin Jul 04 '20 at 03:25
  • By recursively, do you mean you want it to find files in sub-directories too? – Shawn Jul 04 '20 at 03:52
  • Similar questions: [How do I grep recursively in files with a certain extension?](https://stackoverflow.com/q/51952546/4518341), [Recursively look for files with a specific extension](https://stackoverflow.com/q/5927369/4518341), [How to find files recursively by file type and copy them to a directory while in ssh?](https://stackoverflow.com/q/18338322/4518341) – wjandrea Jul 04 '20 at 03:56

2 Answers2

11

Use globstar:

shopt -s globstar
for f in ./**; do
    touch "$f"
done

From the Bash manual:

globstar

If set, the pattern ‘**’ used in a filename expansion context will match all files and zero or more directories and subdirectories. If the pattern is followed by a ‘/’, only directories and subdirectories match.

BTW, always quote your variables, and use ./ in case of filenames that look like options.

This is based on codeforester's answer here

wjandrea
  • 28,235
  • 9
  • 60
  • 81
4

When you say bash do you really mean within bash or just scripting using common command-line tools?

Scripting tools together

find . -type f -exec chmod u+x {} \;

What this means is for every file in the current directory make it executable. The \; is passing the ; to the find command which interprets it as "invoke the exec string on each found path individually". You can replace \; with \+ which tells find to first gather all paths & substitute it for {} all at once. Generally \+ can be more efficient but you have to be careful with command-line lengths as there are limits. What you can do then is combine it with xargs:

find . -type f -print0 | xargs -0 -P $(nproc) -I{} chmod u+x {}

What this does is it tells find to use the null character as a terminator instead of newlines. This ensures that you process each entry correct even if it has arbitrary spaces or random UTF characters (\0 is not a valid part of a path). The -0 option to xargs tells it to use \0 as the separator when reading arguments instead of newliens. The -P option says to run the command in parallel N times where in this case N is the output of the nproc command which prints the number of processors. -I is the substitution string and the rest is the command string to process.

The man pages for find & xargs are good to explore.

Natively within Bash

On the off-chance you're looking for a solution wholly within Bash & no external tools, it's a bit more complicated & would involve some more advanced Bash-specific language constructs where you implement find yourself. To iterate over the contents of a directory, you'd do something like for path in <dir>; do. Then you'd use the test built-in [[ -d "$path" ]] to determine if it's a directory, [[ -f "$path" ]] if it's a file etc (man test has many of the explanations but note that's the standalone test executable which has subtle differences & pitfalls from the more feature-filled & safer bash version [[ ]].

Working with bash arrays: https://www.tldp.org/LDP/Bash-Beginners-Guide/html/sect_10_02.html Bash test introduction: https://www.tldp.org/LDP/abs/html/testconstructs.html

What that test introduction doesn't mention is things like regular expressions which would be part of that syntax. Bash also has powerful options for manipulating the contents of variables: https://www.tldp.org/LDP/abs/html/parameter-substitution.html

In practice though, anything even moderately complex (whether within bash or by combining tools), is probably better maintained & easier to read in Python (speaking as someone with lots and lots of extensive experience in Bash).

find_files() {
  if [[ ! -x "$1" ]]; then
     echo "$1 isn't a directory" >&2
     return 1
  fi

  local dirs=("$1")

  while [[ "${#dirs[@]}" -gt 0 ]]; do
    local dir="${dirs[0]}"
    dirs=("${dirs[@]:1}") # pop the element from above

    # whitespace in filenames iterated will be a problem. Look to the IFS
    # variable to handle those more gracefully.
    for p in "${dir}"/*; do 
      if [[ -d "$p" ]]; then
         dirs+=("$p")
      elif [[ -f "$p" ]]; then
         echo "$p"
      fi
    done
  done
}

for f in $(find_files .); do
    chmod u+x "$f"
done

As you can see this is more complicated, trickier, & slower than just using the find/xargs binaries. You would never want to write something like that in reality. You could even get fancier where you convert find_files into process by having it take a command that you then evaluate as you're iterating (instead of echo'ing the path) via eval. eval is super tricky & can be a security exploit.

Vitali
  • 3,411
  • 2
  • 24
  • 25