How to append data to a nested JSON file in Python

Question

I'm creating a program that will need to store different objects in a logical structure on a file which will be read by a web server and displayed to users.

Since the file will contain a lot of information, loading the whole file tom memory, appending information and writing the whole file back to the filesystem - as some answers stated - will prove problematic. I'm looking for something of this sort:

foods = [{
"fruits":{
    "apple":"red",
    "banana":"yellow",
    "kiwi":"green"
}
"vegetables":{
    "cucumber":"green",
    "tomato":"red",
    "lettuce":"green"
}
}]

I would like to be able to add additional data to the table like so:

newFruit = {"cherry":"red"}
foods["fruits"].append(newFruit)

Is there any way to do this in python with JSON without loading the whole file?

Cesar Canassa · Accepted Answer · 2019-01-17 13:15:50Z

1

That is not possible with pure JSON, appending to a JSON list will always require reading the whole file into memory.

But you could use JSON Lines for that. It's a format where each line in a valid JSON on itself, that's what AWS uses for their API's. Your vegetables.json could be written like this:

{"cucumber":"green"}
{"tomato":"red"}
{"lettuce":"green"}

Therefore, adding a new entry is very easy because it becomes just appending a new entry to the end of the file.

edited Jan 17, 2019 at 13:15

answered Jan 17, 2019 at 11:30

Cesar Canassa

20.5k13 gold badges68 silver badges70 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

bruno desthuilliers · Accepted Answer · 2019-01-17 11:09:10Z

Since the file will contain a lot of information, loading the whole file tom memory, appending information and writing the whole file back to the filesystem - as some answers stated - will prove problematic

If your file is really too huge to fit in memory then either the source json should have been splitted in smaller independant parts or it's just not a proper use case for json. IOW what you have in this case is a design issue, not a coding one.

There's at least one streaming json parser that might or not allow you to solve the issue, depending on the source data structure and the effective updates you have to do.

This being said, given today's computers, you need a really huge json file to end up eating all your ram so before anything else you should probably just check the effective file size and how much memory it needs to be parsed to Python.

Collectives™ on Stack Overflow

How to append data to a nested JSON file in Python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related