Iterating through all folders of an undefined amount of subdirectories

Question

I am in need of a way to script an iteration through an undefined amount of sub-directories.

I'm using kornshell to and awk. I've stripped the problem to this:

for file in $FILEPATH/*.txt
do
    awk '
        { 
            #some awk commands
        }
    '
done

So I need to change $FILEPATH/*.txt so it drills into directories until the end, then the loop iterates through all text files, backs out a directory and iterates through text files once more until no more directories or text files were untouched.

For context, I need to parse through a database and run some commands on certain files. So these file structures could go on for a while.

I'm unsure how to use it for exactly what I need. The undefined amount of sub-directories is what is so weird about this. — Ausche, Mar 09 '18 at 15:39
That is what find is made for. https://www.gnu.org/software/findutils/manual/html_mono/find.html Take the time to read and experiment. — Yunnosch, Mar 09 '18 at 15:40
`find` syntax is weird and picky. `find "$FILEPATH/" -type f -name *.txt`. Not sure about ksh syntax of for-loop, but in Bash it would be `for n in $(find "$FILEPATH/" -type f -name *.txt);do`...Note the trailing slash on FILEPATH, though it could be part of the variable value instead. — LinuxDisciple, Mar 09 '18 at 15:48
Okay thank you both, I've been messing with find. I think I'm close to getting it right, just need to tie everything together. — Ausche, Mar 09 '18 at 16:12
Possible duplicate: [How to loop through a directory recursively to delete files with certain extensions](https://stackoverflow.com/q/4638874/45249) — mouviciel, Mar 09 '18 at 16:55

score 1 · Accepted Answer · answered Mar 09 '18 at 16:53

1

modify your script like below it will work for your (in place of . put your path like /tmp/my/personal/. This is from where you want to search files:-

 FILEPATH=$(find . -name "*.txt")
 for file in $FILEPATH
 do
    echo $file
    awk '
    { 
        #some awk commands
    }
'

 done

answered Mar 09 '18 at 16:53

Abhijit Pritam Dutta

5,521
2
11
17

This will fail if files have space, tab, or newline in their name. For example: `one two.txt` will be treated like two files, `one` and `two.txt`, due to word splitting. – codeforester Mar 10 '18 at 21:24
Added [my answer](https://stackoverflow.com/a/49214202/6862601) that handles the word splitting issues adequately. – codeforester Mar 11 '18 at 01:08

score 1 · Answer 2 · answered Mar 10 '18 at 20:54

To avoid struggling with filenames including blank (or other disturbing things) See Bash Pitfalls #1

find . -name "*.txt" -type f -exec awk '{ ... }' {}  \;

. start the search in the actual directory
-name limit the search to a pattern
-type limit the search to files
-exec execute a command for each file found
{} insert the filename here
\; terminate the command. Escaped, so that the shell does hand it over to find

score 0 · Answer 3 · answered Mar 10 '18 at 21:27

The simplest way to do this in Bash is to turn on globstar option and use a glob to loop around:

#!/bin/bash

cd "$FILEPATH" || { printf '%s\n' "Cant' change directory to '$FILEPATH'"
shopt -s globstar
for file in **/*.txt; do
  # the awk code
  awk '...' "$file"
done

This is immune to world splitting, i.e., it will work fine even if there are whitespace characters in file names.

If you have to use find to do this, then:

while read -d '' -r file; do
  awk '...' "$file"
done < <(find "$FILEPATH" -name '*.txt' -print0)

Iterating through all folders of an undefined amount of subdirectories

3 Answers3