3

I am trying to merging hundreds of thousand JSON files that have a single object. with the help of this answer I have successfully added all objects into a single object as an array. Now I want to filter out some objects during merge if a specific field or parameter is duplicate. Something like field 1 of two objects have the same value.

Here is the example

Input File 1: 

[
    {
      "field 1": 10,
      "field 2": 25,
      "field 3": 35,
      "field 4": 45
    }
]

Input file 2:

[
    {
      "field 1": 15,
      "field 2": 25,
      "field 3": 35,
      "field 4": 45
    }
]

Input file 3:

[
    {
      "field 1": 10,
      "field 2": 20,
      "field 3": 30,
      "field 4": 40
    }
]

Expected output:

[
    {
      "field 1": 10,
      "field 2": 25,
      "field 3": 35,
      "field 4": 45
    },
    {
      "field 1": 15,
      "field 2": 25,
      "field 3": 35,
      "field 4": 45
    }
]

The file 3 object will not merge due to having field 1 value same

Improve this question
0

1 Answer 1

Reset to default
5

You just use the unique_by() function in jq which expects an array of objects and takes the field name as the input to remove duplicates on

jq -n '[ inputs[] ] | unique_by(."value 1")'

To append on top of my other answer to uniquely select based on one field and remove objects by another field you can do below. The below code groups objects excluding duplicates based on "value 1" and excludes ones containing "value 6"

jq -n '[ [inputs[]] | unique_by(."value 1")[] | select( has("value 6") | not )]  '
Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.