How to wait for all spawned and backgrounded processes to finish in bash script

If you have no other processes you are waiting for, a wait with no arguments will wait for all background jobs to finish. I often use constructs like

for i in *; do md5sum $i & done
wait

This will however break if your script has any other background tasks, but you can wrap it in a subshell like

(for i in *; do md5sum $i & done; wait) & all_md5sum=$!
...
wait $all_md5sum

First try a simpler scenario such as only 2 processes FOO1 and FOO2, and if you were to run within a script for example named parent.sh:

#!/bin/bash

FOO1 &
pid1=$!

echo "Child PID=$pid1 launched"

FOO2 &
pid2=$!

echo "Child PID=$pid2 launched"

BAR exit FOO1
BAR exit FOO2

echo "Pausing to wait for children to finish..."

wait $pid1
wait $pid2

echo "Children finished, moving on."

See if this works with the two FOOs and if so, then implement for the 20.

Explanation

Since we are unable to see the internals of FOOs and BAR process, I am relying only on what you've posted and, as I understand it, you said

• "Process BAR is used to check FOO and is also used to kill it"
• meaning, at some point prior to your posted snippet, you started FOO1 somehow, in order for BAR exit FOO1 to affect it

From among your originally posted snippet, you also wrote:

BAR exit FOO1
PID1=$!

• $! captures the most recently background-ed process
• but BAR exit FOO1 is not a process going into the background, it is as you claim, a means of using BAR to control FOO: "BAR is used to check FOO and is also used to kill it"
• thus $! is unlikely to be capturing FOO1's pid

So instead, ensure that right where you actually start a FOO* process, you capture pid. For example if you are running all in parent.sh, and have only 2 processes, you would do:

#!/bin/sh

FOO1 &
pid1=$!

echo "Child PID=$pid1 launched"

FOO2 &
pid2=$!

echo "Child PID=$pid2 launched"

• lowercase pid1, you can use your original uppercase too, as long as consistent, I just prefer to use lowercase as a convention to differentiate from being mistaken as environment variables
• The echo is an optional trace so we know what is going on, use it while we are troubleshooting and comment it out with # or delete it once you've solved this problem

We then do the commands you claimed you use to check and kill FOO1 and FOO2:

BAR exit FOO1
BAR exit FOO2

Then,

echo "Pausing to wait for children to finish..."

wait $pid1
wait $pid2

echo "Children finished, moving on."

• again the echos are optional but informative while you are still troubleshooting this, lets us know what is happening
• wait for each individual $pid...
• followed by another optional echo to let you know the waiting is over

CREDIT

Shotts 2009 asynchronous scripts example on the free pdf's page 506

You could touch a file for each command and check on that file. Here is a simple example:

#!/usr/local/bin/bash

# Flag to knkow when completed
finished=0;

# The commands t run
declare -a commands=('cmd1' 'cmd2' 'cmd3' 'cmd4');

# Fork all commands
for cmd in "${commands[@]}"; do
        ./$cmd &
done

# Loop to wait for all processes to finish
loop=0;
while [ $finished -eq 0 ]; do
        # Just for visual effect
        loop=$((loop+1));
        echo "Loop $loop";
        sleep 1;

        files=0;
        for cmd in "${commands[@]}"; do
                # Each program shoudl touch a file so that we
                # can know it has completed successfully
                if [ -e ".$cmd" ]; then
                        echo "Process $cmd completed";
                        files=$((files+1));
                else
                        # No need to continue if anything is not done
                        # Error checking here like a TTL value or ps -ef | grep pid etc
                        echo "Waiting for process $cmd";
                        break;
                fi;
        done

        # if we have as many files as executed commands
        if [ "$files" -eq ${#commands[@]} ]; then
                finished=1;
        fi
done;

echo 'All jobs completed successfully';
# Clean up after yourself
for cmd in "${commands[@]}"; do
        rm -f ".$cmd";
done;

Then for testing each cmd1, cmd2 ... cmdN can have something as follows:

#!/usr/local/bin/bash
num=1;

echo "Enter cmd$num";
sleep 1; # use different sleep values to simulate execution
touch ".cmd$num"
echo "Exit cmd$num";

This will evaluate to true if the $PID1 is running: if [kill -s 0 $PID1 ]

You could have multiple if clauses. They will evaluate to false if the PID isn't running. This clause (combined with others) should control the flow of the script as you desire.