4

I am executing below script

LOGDIR=~/curl_result_$(date |tr ' :' '_')

mkdir $LOGDIR

for THREADNO in $(seq 20)
do
for REQNO in $(seq 20)
do
 time curl --verbose -sS  http://dummy.restapiexample.com/api/v1/create --trace-ascii ${LOGDIR}/trace_${THREADNO}_${REQNO} -d @- <<EOF >> ${LOGDIR}/response_${THREADNO} 2>&1
 {"name":"emp_${THREADNO}_${REQNO}_$(date |tr ' :' '_')","salary":"$(echo $RANDOM%100000|bc)","age":"$(echo $RANDOM%100000|bc)"}
EOF
echo -e "\n-------------------------------" >> ${LOGDIR}/response_${THREADNO}
done 2>&1 | grep real > $LOGDIR/timing_${THREADNO} &
done

After sometime if i check for no of bash processes, it shows 20(not 1 or 21)

ps|grep bash|wc -l

The question is since I have not used brackets "()" to enclose inner loop, new shell process should not be spawned. I want to avoid creating new shells as the CPU usage nears 100%. I don't know if it matters, but i am using Cygwin.

Improve this question

2 Answers 2

Reset to default
7

Because you have piped the loop into grep, it must be run in a subshell. This is mentioned in the Bash manual:

Each command in a pipeline is executed in its own subshell, which is a separate process (see Command Execution Environment)

It is possible to avoid that with the lastpipe shell option for the final command in the pipeline, but not any of the others. In any case, you've put the whole pipeline into the background, which also creates a subshell.

There is no way around this. What you're doing inherently does require separate shell processes in order to work: even ignoring the pipeline, creating a background process requires creating a process.

If your issue is the CPU usage, that's caused by running everything at once. If you remove the & after the grep, all the commands will run in sequence instead of simultaneously. There will still be subshells created (for the pipeline), but those are not themselves the main issue in that case. If you need them to run simultaneously, the increased CPU usage is the trade-off you've chosen.

Improve this answer
1
  • i removed pipe to grep and executed the script once more. But still 20 shells were spawned. I think that just for executing for loop in background new shell process is required. thanks Commented Apr 28, 2019 at 15:25
0

What won't work

Bash does not support threading parallelism, only multi-process parallelism.

Bash has no way of running a for loop (or a pipe, for that matter) in the background without spawning a child process. It is surprising to me that there are 20 bash processes rather than 21.

I know nothing about Cygwin.

Alternatives

If you are somewhat familiar with Python, I suggest you use the Plumbum library to do your call. Python supports threading and it will make everything easier.

Here's your your code, rewritten and tested:

from datetime import datetime
import json
import random
from plumbum import cmd as c
from threading import Thread

def now():
    return datetime.now().strftime("%Y-%m-%d %H_%M_%S")

logdir = f"~/curl_result_{now()}"

def curl(threadno, reqno):
    args1 = "--verbose -sS http://dummy.restapiexample.com/api/v1/create --trace-ascii".split()
    args2 = [f"{logdir}_{threadno}_{reqno}", "-d", "@-"]
    content = json.dumps({
        "name": f"{logdir}/trace_{threadno}_{reqno}_{now()}",
        "salary": random.randrange(100_000),
        "age": random.randrange(100_000) ,
    })
    call = c.echo[content] | c.curl[(*args1, *args2)] >> f"{logdir}/response_{threadno}"
    print(call)
    # call()

def curl_batch(threadno):
    for reqno in range(20):
        curl(threadno, reqno)

# Start 20 threads
threadList = []
for threadno in range(20):
    t = Thread(target=curl_batch, args=(threadno,))
    t.start()
    threadList.append(t)

# Wait for every thread
for thread in threadList:
    thread.join()

Enjoy Python's flexibilty ;)

Improve this answer
3
  • Is this actually invoking external curl processes? Why not do the HTTP requests within the Python runtime? Commented Apr 28, 2019 at 14:36
  • Indeed, it uses the curl command. I guess I could use import urllib.request as req; req.urlopen() too, plubmum is not mandatory. But I can't guarantee the behavior of urlopen() to be the same as that of curl. Besides, OP uses a certain number of switches on curl. Reproduce the associated behavior in Python might be a useless effort. Commented Apr 28, 2019 at 21:14
  • There is a curl module for Python (that directly uses libcurl), so such a guarantee is entirely possible. Commented Apr 28, 2019 at 21:16

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.