1

The following file runs, but not doing anything, yet it does not error.... 

while read dates; do ./avg_hrly_all_final.sh ${dates}; done < ./dates_all.csv

I have a list of dates in "dates_all.csv" that have the following form:

2005 01
2005 02
2005 03

And the script I am calling "avg_hrly_all_final.sh" works by passing it 2 positional parameters, example:

./avg_hrly_all_final.sh 2005 01

FOLLOW-UP

xargs -n2 ./the_script.sh <./dates_to_pass.csv

OR

while read dates; do ./the_script.sh ${dates}; done <./dates_to_pass.csv

work, just make sure the dates of the file being passed is of the same "End of Line" variety as the machine you are running the command on expects ;)

Improve this question
3
  • As written above, your script should run ./avg_hrly_all_final.sh with two parameters. "no dice" So what actually happened when you ran the script? What made you think that the script's run was not successful? What error message did you see? Commented Mar 1, 2015 at 8:20
  • The script that is supposed to be executed produces a set of files that did not populate. "No dice" is a colloquial term for "it didn't work". No error message. Commented Mar 1, 2015 at 9:37
  • 1
    If using zsh, you need $=dates so $dates is split into 2 arguments. Other Bourne-like shells should do that implicitly unless you modified $IFS Commented Mar 1, 2015 at 10:49

2 Answers 2

Reset to default
3

This is a likely job for xargs:

printf %s\\n '#!/bin/sh' 'printf "<%s>\n" "$$" "$@"' >avg_hourly.sh
chmod +x ./avg_hourly.sh
xargs -n2 ./avg_hourly.sh <<\IN
2005 01
2005 02
2005 03
IN

xargs will split on the spaces by default and invoke the specified command once per -n2 occurring arguments. I just wrote a little dummy avg_hourly.sh script there which prints its arguments one per line as delimited at either end by < and > following its PID in the same format. The above prints:

<1115>
<2005>
<01>
<1116>
<2005>
<02>
<1117>
<2005>
<03>

...just to demonstrate. You should use <./dates_all.csv rather than my <<\IN here-document as input, though, probably.

Improve this answer
0
1

If you have GNU Parallel you can do this:

parallel --colsep ' ' ./avg_hrly_all_final.sh {1} {2} :::: ./dates_all.csv

If the first line of the csv file is the header you can do something like:

parallel --header : --colsep ' ' ./avg_hrly_all_final.sh {Year} {Month} :::: ./dates_all.csv

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.