2

I have a file:

50102.5924   4.2599   4.2184  1.0098   4.2392
50103.5903   4.2895   4.2474  1.0099   4.2685
50107.5850   4.2100   4.2286  0.9956   4.2193
50108.5331   4.1477   4.1112  1.0089   4.1295
50108.7620   4.0770   4.1060  0.9929   4.0915
50109.5345   4.2227   4.2153  1.0018   4.2190
50109.7681   4.1677   4.1673  1.0001   4.1675
50110.5308   4.2333   4.3158  0.9809   4.2746
50110.7612   4.2339   4.2743  0.9905   4.2541
50111.5591   4.1330   4.1542  0.9949   4.1436
50112.5324   4.1417   4.0986  1.0105   4.1202
50112.7668   4.0075   3.9844  1.0058   3.9960
50113.5301   4.2147   4.2147  1.0000   4.2147
50113.7639   4.2263   4.2263  1.0000   4.2263
50114.5321   4.1205   4.1211  0.9999   4.1208

And many files:

4.5149 50102.5924   72.220     1.000     1    1
4.5683 50103.5903   -3.800     1.000     1    1
4.4682 50107.5850  -23.670     1.000     1    1

How to replace the first column in many files by the last column of the file in such a way that the first column of the file is the same as the second column of many files.

The desired result for the small file given for an example is

4.2392 50102.5924   72.220     1.000     1    1
4.2685 50103.5903   -3.800     1.000     1    1
4.2193 50107.5850  -23.670     1.000     1    1

I tried:

for f in small_file*; do 
    awk 'NR==FNR{ar[$1]=$5;next} ($2 in ar) {$1= ar[$1]}1'  her_OK "$f" > "${f}_em"
done

The first columns of small files disappered instead of to be replaced.

Improve this question
2
  • 3
    ($2 in ar) {$1= ar[$1]} should be ($2 in ar) {$1= ar[$2]} since it is $2 and not $1 that is the index. Is that the only issue? Commented Jul 20, 2021 at 14:49
  • Do you need to retain the white space or is it OK if that gets changed by the tool (e.g. multiple blanks or a tab, whatever you have separating fields, become a single blank or tab)? Commented Jul 20, 2021 at 15:33

1 Answer 1

Reset to default
5

You should be using $1 = ar[$2], not ar[$1].

Also, you don't need the for loop, or to do the redirection in shell. awk can construct filenames and redirect its output by itself.

awk 'NR==FNR {
       ar[$1]=$5; next
     }

     ($2 in ar) {
       $1 = ar[$2];
       print > FILENAME "_em"
     }'  her_OK small_file*

Output:

4.2392 50102.5924 72.220 1.000 1 1
4.2685 50103.5903 -3.800 1.000 1 1
4.2193 50107.5850 -23.670 1.000 1 1

BTW, you probably should use a different filename prefix (or a different output directory) so any subsequent runs don't re-process the small_file*_em files too. e.g.

mkdir new

and change the print statement in the awk script to:

print > "new/" FILENAME "_em"
Improve this answer
5
  • when writing to more than hundred files within awk, I sometime stumble upon a limitation. Commented Jul 20, 2021 at 15:06
  • @Archemar In GNU awk you can use ENDFILE { close(FILENAME) }...or for this answer: ENDFILE { close("new/" FILENAME "_em")}. That stops awk from keeping output files open when they're no longer needed. Commented Jul 20, 2021 at 15:17
  • In GNU awk you don't need to close the files as you go, gawk handles opening/closing the FDs as they're needed. There is a performance hit for that though as the number of "open" files grows large. Commented Jul 20, 2021 at 15:26
  • print > FILENAME "_em" or any other unparenthesized expression on the right side of output or input redirection is undefined behavior per POSIX, it has to be print > (FILENAME "_em") to work in all awks, otherwise you'll get a syntax error from some of them, e.g. the BSD awk on MacOS IIRC. Commented Jul 20, 2021 at 15:26
  • Make it FNR==1 { close(out); out=FILENAME "_em" } ... print > out and then it'll work robustly and efficiently in every awk. Commented Jul 20, 2021 at 15:28

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.