execute bash commands sequentially inside for loop in Python

Question

I am trying to run 2 bash commands sequentially in a for loop in Python. The loop is as given below:

for dataset in data:
        subprocess.call("cd " +dataset+'.matrix',shell=True)
        subprocess.call("cat part-r-* > combined_output", shell=True)

However, in this way, each command is being considered independently of each other. I need them to execute one after the other. I don't know how to use the subprocess module well (I also tried with os.system). I went through some documentation online but they weren't really useful. Any help in this would be appreciated. thanks in advance!

etheranger · Accepted Answer · 2014-07-08 09:02:50Z

3

I don't believe they're running asynchronously (at the same time), just that they're in different subprocesses which each inherit the parent process' working directory. So it is waiting for the cd to complete, but the subprocess which has cd'd then disappears.

Thankfully, in this case you can tell the call to operate in a different directory with the cwd parameter, so you don't need your cd:

for dataset in data:
    subprocess.call("cat part-r-* > combined_output",shell=True,cwd=dataset+'.matrix')

answered Jul 8, 2014 at 9:02

etheranger

1,2937 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

perreal · Accepted Answer · 2014-07-08 08:54:50Z

0

An example with os.system:

import os
os.system("echo aaa > ~/SOMEFILE1111; cd ~/; cat SOMEFILE1111; rm SOMEFILE1111")

and one with subprocess.call:

import subprocess
cmds = [ "echo aaa > ~/SOMEFILE1111", "cd ~/",
        "cat SOMEFILE1111", "rm SOMEFILE1111"]
subprocess.call(";".join(cmds), shell=True)

edited Jul 8, 2014 at 8:54

answered Jul 8, 2014 at 8:48

perreal

98.7k23 gold badges159 silver badges187 bronze badges

2 Comments

user3033194 Over a year ago

is it OK if I write like this: os.system("cd " +dataset+ '.matrix';"cat part-r-* > combined_output")

perreal Over a year ago

It should be like this: "cd " +dataset+ '.matrix; cat part-r-* > combined_output'

Collectives™ on Stack Overflow

execute bash commands sequentially inside for loop in Python

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related