Read JSON data from API to Python

Question

I am trying to read from an API data on component setups. Basically the communication with the API works. However, I noticed that not all values I expect are loaded.

There should be 52 components (https://lignumdata.ch/?page=bauteil&bauteilgruppe=decke) in the list, but I get back only 13.

API = https://lignumdata.ch/api/

what am I doing wrong?

import json
import urllib.request
import urllib.parse
import urllib



def main():
    URL = 'https://lignumdata.ch/api/v1.cfc?method=getBauteil&compatMode=true&condition={"type":"decke","bauteiltyp":["Rippen", "Balken"]}'
    
request_url = URL.replace(" ", "")
    
    print('url', request_url)

    response = urllib.request.urlopen(request_url)
    data = json.loads(response.read()) 
    #print('url', data[0])

    for output_loads in data:
        print(output_loads['laufnummer'])
                
    return


if __name__ == '__main__':
    main()

Samy · Accepted Answer · 2021-03-12 23:07:56Z

1

The results are paginated. You need to pick a specific page with page={pagenum}. However, the API you provided mentioned this: If authenticated, the getBauteilIndex route now accepts page=all to turn off pagination.

I don't know if it's possible to turn pagination off for the method you used though, but even if it is possible, you will need to be authenticated.

EDIT:

Based on the comments, here's how to send an authenticated request:

import request
import base64

url = "http://example.com"

username = "user"    
password = "password"
authstr = username + ':' + password
asciistr = authstr.encode('ascii')
b64str = base64.b64encode(asciistr).decode()

headers = {
    'Authorization': "Basic " + b64str
    }

querystring = {"param1": "value", "param2": "value"}
response = requests.request("GET", url, headers=headers, params=querystring)

print(response.text)

edited Mar 12, 2021 at 23:07

answered Mar 12, 2021 at 18:38

Samy

6598 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

chrisbyte Over a year ago

You beat me to it. The conditions the OP are using only has 13 results regardless, but if I remove "bauteiltyp" and only search for "type":"decke" I get the 13 pages of results with 20 items per page, which are the same results from the link OP posted, which is 20 items * 13 pages = 260 results

chrisbyte Over a year ago

Here is the section on paging to add headers to get all the data lignumdata.ch/api/#paging

Samy Over a year ago

Exactly! If they allowed unauthenticated non-paginated results, I would say they have a terrible backend.

Michael Brunner Over a year ago

Thanks for your help @chrisbyte! Somehow I can not get this to work in my URL, how and where do I have to put the paging? A password I can request, how would I have to create the script so that my approach would work?

Samy Over a year ago

@MichaelBrunner It's right there in the API: lignumdata.ch/api/#toc_110 Just add "Authorization" header with the value of "basic " followed by the base64 encoding of "username:password".

|

Collectives™ on Stack Overflow

Read JSON data from API to Python

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related