0

I am trying to import data from an API and later export it into a CSV file. However this it not working and I'm getting the following error: 

expected string or buffer.

I even tried using json.dumps instead of json.load, I get the following error: 

<response 200 is not json serializable.

Sample code:

import requests
from requests.auth import HTTPBasicAuth
import pandas as pd
import json
import csv


proxies = {
    'http': 'http://dummy.restapiexample.com/api/v1/employees
    'https': 'http://dummy.restapiexample.com/api/v1/employees
}

url = 'http://dummy.restapiexample.com/api/v1/employees' 
r = s.get(url=url, proxies=proxies,  auth=HTTPBasicAuth('user', 'pass'))
employee_parsed = json.loads(r) 
emp_data = employee_parsed['Employee ID']
employ_data = open('"Path" testname.csv', 'w') 
csvwriter = csv.writer(employ_data)
count = 0

for emp in emp_data:
      if count == 0:
             header = emp.keys()
             csvwriter.writerow(header)
             count += 1

      csvwriter.writerow(emp.values())

employ_data.close()

Does anyone have an idea how I can solve the errors occurring? Any tips would be appreciated or any insight or where to look/think. Thanks!

Martin Evans
  • 45,791
  • 17
  • 81
  • 97
Jordan
  • 657
  • 3
  • 9
  • 16

3 Answers3

1

Change this:

employee_parsed = json.loads(r)

to this:

employee_parsed = json.load(r)

Check out this answer, which talks about the difference between json.loads and json.load.

Edit:

As per @roganjosh you can use the builtin json decoder, something like:

r = s.get(url=url, proxies=proxies, auth=HTTPBasicAuth('user', 'pass')).json()

Check out the description here.

R4444
  • 2,016
  • 2
  • 19
  • 30
1

There are multiple problems with your sample code. I have corrected the code to read the api and deliver the csv as output.

import requests as s
from requests.auth import HTTPBasicAuth
import pandas as pd
import json
import csv

proxies = {
    'http': 'http://dummy.restapiexample.com/api/v1/employees',
    'https': 'http://dummy.restapiexample.com/api/v1/employees'
}

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'}
url = 'http://dummy.restapiexample.com/api/v1/employees' #get key

r = s.get(url=url, headers=headers)

employee_parsed = r.json();
emp_data = employee_parsed;

employ_data = open('testname.csv', 'w', encoding="utf-8") 

csvwriter = csv.writer(employ_data)

count = 0

for emp in emp_data:

      if count == 0:

             header = emp.keys()

             csvwriter.writerow(header)

             count += 1

      csvwriter.writerow(emp.values())

employ_data.close()

You can use response.json() to directly get the json response as shown in my example. If you want to parse the response using json.loads then you have to parse only the content of the response. This can be done as follows,

employee_parsed = json.loads(r.content)

The primary problem was with the API used not returning a proper response unless the user agent is spoofed by Python. This is solved by inserting a spoofed header.

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'}
url = 'http://dummy.restapiexample.com/api/v1/employees' #get key

r = s.get(url=url, headers=headers)

I imported requests as s in my example to ensure the code executes.

trance
  • 128
  • 1
  • 8
  • Hi, thank you for this. Tried it with different variations but I get Attributterror: 'dict' object has no attribute 'content' If I use json.loads(r.text) it's the same thing but ..no attribute 'text' etc Also tried with load and loads, same error – Jordan Jul 08 '19 at 20:51
  • @Jordan If you run the above code as is, you should not get any errors. From the error message you point out looks like r is a dict object. Put the following snippet after the statement where you make the call. **print(type(r))** You should see something like this in console, **** – trance Jul 09 '19 at 01:11
1

You should extract text from your response employee_parsed = json.loads(r.text) or just employee_parsed = r.json()

Bohdana
  • 11
  • 1