How can I read/store an array in bash in parallel?

Question

I have already read posts like How can I store the “find” command results as an array in Bash or Creating an array from a text file in Bash or Store output of command into the array

Now my issue is the following: How to do this in parallel?

Background:

I have a script for processing a large git repository with a lot of submodules and perform certain actions within these. Sometimes there are some tasks that take a while so meanwhile I want to give some user feedback to indicate that something is still happening and the code isn't just stuck ^^

I have a function

function ShowSpinner()
{
    pid=$!
    while [ -d /proc/$pid ]
    do
        for x in '-' '/' '|' '\\'
        do
            echo -ne ${x}" \r"
            sleep 0.1
        done
    done
}

for displaying a little spinner while doing long tasks. And so far currently I use this e.g. like

while IFS= read -r line
do
    # Some further processing of the output lines here
done <<< $(git pull 2>&1) & ShowSpinner

which works fine and always displays the spinner until the task is finished.

In particular I use this also for finding submodules in a git repository like

function FindSubmodules()
{
    # find all .git FILES and write the result to the temporary file .submodules
    find -name ".git" -type f > .submodules & ShowSpinner
    # read in the temporary file
    SUBMODULES=$(cat .submodules)
    # and delete the temporary file
    rm .submodules
}

later I iterate the submodules using e.g.

function DoSomethingWith()
{
    for submodule in ${SUBMODULES}
    do
        echo $submodule
    done
}

FindSubmodules

DoSomethingWith

Of course I do more stuff in there, this is only a short example.

This works find, but what I don't like here is that this file .submodules is created (and if only temporary). I would prefer to directly store the result in an array and then iterate that one directly.

So after reading mentioned posts I tried to use something like simply

IFS=$'\n'
SUBMODULES=( $(find -name ".git" -type f)) & ShowSpinner

or from the links also

readarray SUBMODULES < <(find -name ".git" -type f) & ShowSpinner

or

readarray -t SUBMODULES "$(find -name ".git" -type f)" & ShowSpinner

and then iterate like

for submodule in ${SUBMODULES [@]}
do
    echo $submodule
done

For all three options the result is basically the same: The spinner works fine but all that I get using this is one single entry with the last char of the ShowSpinner instead of the results of find. Without the & ShowSpinner it works fine but of course doesn't show any feedback of a long tasks.

What am I doing wrong? How can I get the readarray to work in parallel with the ShowSpinner function?

---

Update as suggested I have put it to a function (actually I already had functions just didn't put the spinner behind the entire function so far)

function FindSubmodules()
{
    echo ""
    echo ${BOLD}"Scanning for Submodules ...  "${NORMAL}
    
    SUBMODULES=($(find -name ".git" -type f))
    
    for submodule in "${SUBMODULES[@]}"
    do
        echo $submodule
    done
}

function CheckAllReposForChanges()
{
    # Check Submodules first
    for submodule in "${SUBMODULES[@]}"
    do
        # remove prefixed '.'
        local removedPrefix=${submodule#.}
        # remove suffix '.git'
        local removedSuffix=${removedPrefix%.git}

        echo "${BASEPATH}${removedSuffix}"
    done
    
    # Check the main repo itself
    echo "${BASEPATH}"

    echo ""
}

FindSubmodules & ShowSpinner

CheckAllReposForChanges

the CheckRepoForChanges function itself works just fine.

What I get now is the spinner and then the correct output from the first FindSubmodules like e.g.

./SomeFolder/.git
./SomeOtherFolder/.git
./SomeThirdFolder/.git

etc

However when it comes to the CheckAllReposForChanges (again the echo is just an example for debugging) I don't get any output except the main repository path. It seems like now SUBMODULES is empty since it is being filled in the background. It worked with the solution I used originally.

Answer 1

Of course the array is empty; you have backgrounded the function which will eventually populate it (and anyway, there is no way really for the background process to populate a variable in its parent once it finishes).

Run both functions in the same background process and they will be able to communicate properly.

{ FindSubmodules
  CheckAllReposForChanges
} & ShowSpinner

Answer 2

Maybe I'm misreading the question but it seems (to me) the requirement is to pass data 'up' from a backgrounded/child process to the calling/parent process.

A backgrounded script/function call spawns a new, asynchronous OS-level process; there is no easy way to pass data 'up' from the child process to the parent process.

While it may be possible to build some sort of inter-process shared memory structure to share data between parent and child processes, it's a bit easier if we can use some sort of intermediate storage (eg, fifo, file, database table, queuing system, etc) that the various processes can 'share'.

One idea:

• parent process creates one or more temp directories (eg, one for each distinct array to be populated)
• each child process writes data to a file (filename = ${BASHPID}) in a particular temp directory in a format that can be easily parsed (and loaded into an array) by the parent
• the parent calls the child process, waits for the child process to complete, and then ...
• the parent process reads the contents of all files in the temporary directory(s) and loads the appropriate array(s)

For sake of an example I'll assume we just need to populate a single array; I'm also going to use the same temp directory for capturing/storing modules for the functions regardless of whether each function is run in the background or foreground:

unset submodules                                     # delete any variable with this name
submodules=()                                        # init array

outdir=$(mktemp -d)                                  # create temp directory

FindSubmodules()
{
    ... snip ...
        echo "$submodule" >> "${outdir}/${BASHPID}"  # write module to temp file
    ... snip ...
}

CheckAllReposForChanges()
{
    ... snip ...
        echo "$submodule" >> "${outdir}/${BASHPID}"  # write module to temp file
    ... snip ...
}

FindSubmodules & ShowSpinner

CheckAllReposForChanges

# now pull modules from temp file(s) into array; NOTE: assumes each temp file contains a single module name on each line, and no blank lines, otherwise OP can add some logic to address a different format

while read -r modname
do
    submodules+=(${modname})
done < <(cat "${outdir}"/[0-9]*)

# remove temp directory and file(s)

'rm' -rf ${outdir}

Answer 3

If you can write your parallelism using GNU Parallel, you can use parset:

dostuff() {
  #  Do real stuff here
  sleep 10;
}
export -f dostuff

function ShowSpinner()
{
    while [ -d /proc/$pid ]
    do
        for x in '-' '/' '|' '\\'
        do
            echo -ne ${x}" \r"
            sleep 0.1
        done
    done
}

sleep 1000000 &
pid=$!
ShowSpinner &
parset myout dostuff < <(find -name ".git" -type f)
kill $pid
echo

Or (if you are willing to change ShowSpinner:

dostuff() {
  #  Do real stuff here
  sleep 10;
}
export -f dostuff

function ShowSpinner()
{
    while true; do
        for x in '-' '/' '|' '\\'
        do
            echo -ne ${x}" \r"
            sleep 0.1
        done
    done
}

ShowSpinner &
pid=$!
parset myout dostuff < <(find -name ".git" -type f)
kill $pid
echo