Loop through CSV file and create new csv file with while read?

Question

I have

while read $field1 $field2 $field3 $field4
do
  $trimmed=$field2 | sed 's/ *$//g'
  echo "$trimmed","$field3" >> new.csv
done < "$FEEDS"/"$DLFILE"

Now the problem is with read I can't make it split fields csv style, can I? See the input csv format below.

I need to get columns 3 and 4 out, stripping the padding from col 2, and I don't need the quotes.

Csv format with col numbers: 12 24(")25(,)26(")/27(Field2values) 42(")/43(,)/44(Field3 decimal values) "Field1_constant_value","Field2values ",Field3,Field4

Field1 is constant and irrelevant. Data is quoted, goes from 2-23 inside the quotes. Field2 fixed with from cols 27-41 inside quotes, with the data at the left and padded by spaces on the right. Field3 is a decimal number with 1,2, or 3 digits before the decimal and 2 after, no padding. Starts at col 74. Field4 is a date and I don't much care about it right now.

Rubens · Accepted Answer · 2017-03-15 13:26:12Z

6

Yes, you can use read; all you've got to do is reset the environment variable IFS -- Internal Field Separator --, so that it won't split lines by its current value (default to whitespace), but by your own delimiter.

Considering an input file "a.csv", with the given contents:

1,2,3,4

2,3,4,5

6,3,2,1

You can do this:

IFS=','
while read f1 f2 f3 f4; do
    echo "fields[$f1 $f2 $f3 $f4]"
done < a.csv

And the output is:

fields[1 2 3 4]

fields[2 3 4 5]

fields[6 3 2 1]

edited Mar 15, 2017 at 13:26

answered Nov 17, 2012 at 20:08

Rubens

14.8k11 gold badges68 silver badges94 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

gniourf_gniourf Over a year ago

while IFS=, read f1 f2 f3 f4; do ... will avoid to set IFS for the whole script.

TheRuss · Accepted Answer · 2012-11-17 20:11:13Z

1

A couple of good starting points for you are here: http://backreference.org/2010/04/17/csv-parsing-with-awk/

answered Nov 17, 2012 at 20:11

TheRuss

3183 silver badges9 bronze badges

Collectives™ on Stack Overflow

Loop through CSV file and create new csv file with while read?

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related