Parse JSON and iterate objects using jq

Question

[ 
{ "id":"1", 
  "metadata":{"version":"1.0","author":"user1"}
},
{ "id":"2", 
  "metadata":{"version":"1.0","author":"user2","timestamp":"2019-02-05"}
},
{ "id":"3", 
  "metadata":{"version":"1.0","author":"user3","price":"10.0"}
}]

I am trying to parse the above json (data1.json) using jq, but I am unable to access the metadata object. I will eventually use the metadata object to insert into a database as a string. It can have any number of fields, the structure is not fixed. This is the script that I'm using.

#!/usr/bin/env bash
id=($(jq '.[] | .id' data1.json | tr -d '"'))
metadata=($(jq '.[] | .metadata' data1.json))

n_id=${#id[@]}
n_meta=${#metadata[@]}

echo $n_id 
echo $n_meta 

for (( i=0; i<n_id; i++ )); do
    echo ${metadata[$i]} 
done

Expected output:

{"version":"1.0","author":"user1"}

{"version":"1.0","author":"user2","timestamp":"2019-02-05"}

{"version":"1.0","author":"user3","price":"10.0"}

What am I doing wrong here? Any help is appreciated.

Shawn · Accepted Answer · 2020-06-17 05:10:15Z

2

All the metadata objects, one per line, not wrapped in an array?

$ jq -c '.[] | .metadata' data1.json
{"version":"1.0","author":"user1"}
{"version":"1.0","author":"user2","timestamp":"2019-02-05"}
{"version":"1.0","author":"user3","price":"10.0"}

But you're already doing just that for your metadata array in your shell code. If you want to print that array one per line:

printf "%s\n" "${metadata[@]}"

To get the lines printed by jq into an array when there are spaces in the object contents:

mapfile -t metadata < <(jq -c '.[] | .metadata' data1.json)

or

while IFS= read -r obj; do metadata+=("$obj"); done < <(jq -c '.[] | .metadata' data1.json)

edited Jun 17, 2020 at 5:10

answered Jun 17, 2020 at 4:30

Shawn

53.9k3 gold badges29 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

prathyusha Over a year ago

Thank you so much! This works!! but I see one small problem, if there is space in one of the values, the object is not stored as expected. For example, if "author":"fname lname" has spaces in all the metadata objects, it breaks off after fname.

Shawn Over a year ago

@prathyusha Added a few ways to populate the array that avoid that problem.

Sonia Hamilton · Accepted Answer · 2020-06-17 04:18:00Z

1

You can do this using jtc:

If the above content is in the file foo.json, you can do:

% <foo.json jtc -tc -w '<metadata>l:'
{ "author": "user1", "version": "1.0" }
{ "author": "user2", "timestamp": "2019-02-05", "version": "1.0" }
{ "author": "user3", "price": "10.0", "version": "1.0" }

-tc means "compact output format"

-w .. is the walk path

<metadata> match on 'metadata', l means label match, : means 'all'

edited Jun 17, 2020 at 4:18

answered Jun 17, 2020 at 4:12

Sonia Hamilton

4,4598 gold badges41 silver badges51 bronze badges

1 Comment

Dmitry Over a year ago

it's better to use -r option to ensure single row output for walks. thus the complete answer would be: while IFS= read -r obj; do metadata+=("$obj"); done < <(jtc -w'<metadata>l:' -r data1.json)

Hieu · Accepted Answer · 2020-06-17 04:37:45Z

0

Why not just filter your data with jq itself?

jq -c '.[] | .metadata' data1.json

Please refer to Shawn's answer as I have not accounted for other data.

edited Jun 17, 2020 at 4:37

answered Jun 17, 2020 at 4:11

Hieu

2672 silver badges7 bronze badges

1 Comment

prathyusha Over a year ago

I do not know the contents of metadata, it can be anything.. so I can't set the fields like you mentioned.

Collectives™ on Stack Overflow

Parse JSON and iterate objects using jq

3 Answers 3

2 Comments

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related