Multiple threading multiple lines of code in bash scripts

Question

Part of a script I have goes through all the IP addresses in the LAN, and checks the associated FQDN.

while [[ $ws -le $total_ip ]]; do
    ip="${ip_range}.${ws}"
    machine=$(timeout 0.3s python3 -c "import socket; print(socket.getfqdn('$ip'))")
    if[[ "$machine" = "$myMachine" ]]; then 
        my_ip=$ip
        break
    fi
done

This works right now, but takes quite a long time to complete. I added a timeout command in front of the python command to make sure it doesn't go on for more than 0.3 seconds, but that still is quite long.

I want to know if I can thread this piece of code to speed up the process:

machine=$(timeout 0.3s python3 -c "import socket; print(socket.getfqdn('$ip'))")
if[[ "$machine" = "$myMachine" ]]; then 
    my_ip=$ip
    break
fi

However, it needs to be native bash, because I wan't to run this on other computers without having to install any additional tools.

Not directly relevant to the question but you never increment ws so this seems like an infinite loop. — tjm3772
– tjm3772, Commented Feb 8, 2023 at 14:33
Also if you're already using python anyways it would probably be much faster to write the logic in a python script and not fork a couple hundred children. — tjm3772
– tjm3772, Commented Feb 8, 2023 at 14:35
Write it in python. because I wan't to run this on other computers without having to install any additional tools. You are literally using python. — KamilCuk
– KamilCuk, Commented Feb 8, 2023 at 15:37
Thank you for replying to my post. The thing with this script is that it needs to become all in one line eventually. I'm not sure if I can do that in python. — Young Fellow
– Young Fellow, Commented Feb 8, 2023 at 15:51
What does 'all in one line' mean? python3 -c '<commands>' "$myMachine" "$ip_range".{0..255} is technically one line. — tjm3772
– tjm3772, Commented Feb 8, 2023 at 16:15

Pear · Accepted Answer · 2023-02-08 17:09:19Z

1

According to the Bash documentation you can wrap your code into {} to group it. If a & is put behind this group, it will be executed asynchronously. to wait for all subprocesses to finish, use wait

Example:

{
    echo "foo"
    sleep 1
    echo "bar"
} &
echo "baz"
wait

Will output:

baz
foo
# and after one second
bar

more information on bash job control can be found here.

edited Feb 8, 2023 at 17:09

answered Feb 8, 2023 at 17:08

Pear

715 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

tjm3772 Over a year ago

This is true and accomplishes a degree of asynchronous behavior, but keep in mind that the backgrounded groups are run in subshells which limits how the parent shell can interact with them. For example foo=bar ; { foo=baz ; } & wait ; echo "$foo" prints bar because changes to variables in the subshell aren't reflected in the parent.

tjm3772 Over a year ago

The impact for OP is that if they populate my_ip in one of these subshells they won't have access to it later - they'd have to find another way to tell the parent shell what IP they found, which adds some additional complexity to their eventual solution.

Young Fellow Over a year ago

echo $ip > /tmp/mymachine

tjm3772 · Accepted Answer · 2023-02-08 16:49:40Z

1

A simple mostly-python solution. This passes the machine name and a range of IPs to check, prints the IP if found, and sets a non-zero exit code if it doesn't find an IP for the given machine name. The total runtime is just a few milliseconds if performance is your goal.

#!/bin/bash
myMachine=$1
ip_range=$2

if my_ip=$( python3 -c '
import sys
import socket
my_machine = sys.argv[1]
for ip in sys.argv[2:]:
  if socket.getfqdn(ip) == my_machine:
    print(ip)
    sys.exit(0)
sys.exit(1)
' "$myMachine" "$ip_range".{0..255} ) ; then
  echo "$myMachine IP: $my_ip"
else
  echo "IP for $myMachine not found."
fi

The timeout in your bash example does not improve performance much if at all because each individual instance of python is probably running very fast. Your performance issues are a result of spawning lots of child processes - fork is a very expensive system call, and you're forcing bash to call it repeatedly by invoking a new instance of python3 on every iteration. Even adding "threading" would not improve performance much, because (again) the bottleneck is not that python is too slow, it's that creating an instance of python is a slow operation. Trying to solve the problem with multithreading is just going to result in very messy and error-prone code with minimal gains in performance.

The solution to this type of problem is normally to reduce the number of child processes you create, which is why I recommend calling python3 only once and letting it do the heavy lifting.

edited Feb 8, 2023 at 16:49

answered Feb 8, 2023 at 16:26

tjm3772

3,2344 silver badges15 bronze badges

2 Comments

Young Fellow Over a year ago

Thank you for proposing an answer to my question. However the main issue was that it takes a long time to go through all 254 IP addresses, my bare for loops takes around 1.5 minutes to complete, while your proposed solution took around 5 minutes.

Young Fellow Over a year ago

Also, I added the timeout because if the IP address is not up the python command will wait for about a second and then exit. But if it is alive, then it doesn't take more than 0.3 seconds to return the FQDN

Collectives™ on Stack Overflow

Multiple threading multiple lines of code in bash scripts

2 Answers 2

3 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related