0

I am new to python. I am trying to read json response from requests and filtering using pandas to save in csv file. This script works and gives me all the data but its throws this error after execution -

I am not able to figure out why its throwing this error ? How can I pass this error ?

Error -

 script.py line 42, in <module>
 df = pd.read_csv("Data_script4.csv")
 File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-    
 packages/pandas/io/parsers.py", line 686, in read_csv
 return _read(filepath_or_buffer, kwds)
 File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-   
 packages/pandas/io/parsers.py", line 458, in _read
 data = parser.read(nrows)
 File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-   
 packages/pandas/io/parsers.py", line 1196, in read
 ret = self._engine.read(nrows)
 File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-  
 packages/pandas/io/parsers.py", line 2155, in read
 data = self._reader.read(nrows)
 File "pandas/_libs/parsers.pyx", line 847, in pandas._libs.parsers.TextReader.read
 File "pandas/_libs/parsers.pyx", line 862, in   
 pandas._libs.parsers.TextReader._read_low_memory
 File "pandas/_libs/parsers.pyx", line 918, in pandas._libs.parsers.TextReader._read_rows
 File "pandas/_libs/parsers.pyx", line 905, in   
 pandas._libs.parsers.TextReader._tokenize_rows
 File "pandas/_libs/parsers.pyx", line 2042, in pandas._libs.parsers.raise_parser_error
 pandas.errors.ParserError: Error tokenizing data. C error: Expected 9 fields in line 53,    
 saw 10

This is my script -

if __name__ == '__main__':

    parser = argparse.ArgumentParser("gets data")
    parser.add_argument("-o" , dest="org", help="org name")
    parser.add_argument("-p" , dest="pat", help="pat value")
    args = parser.parse_args()
    org  = args.org
    token = args.pat

    url = "https://dev.azure.com/{org_name}/_apis/git/repositories?  
    api- 
    version=6.0".format(org_name=org)

    data = getproject(url,token) 
    data_file=open("Data_script4.csv", "w",newline='')
    val=data['value']
    csv_writer = csv.writer(data_file)


    for i in val:
        if count==0:
           header=i.keys()
           csv_writer.writerow(header)
           count +=1
        
    csv_writer.writerow(i.values())
    
    pro_name=[]
    time=[]

    df = pd.read_csv("Data_script4.csv")
    for i in df["project"]:
    res = ast.literal_eval(i) 
    pro_name.append(res['name'])
    time.append(res['lastUpdateTime'])

    del df["project"]
    df["project name"] = pro_name
    df["lastUpdateTime"] = time


    df =df[["id","name","url","project   
    name","lastUpdateTime","defaultBranch","size","remoteUrl","sshUrl","webUrl"]]
    df.head()
    df.to_csv("Data_Filtered.csv",index=False)

    print("\nFile Created Successfully...")
    data_file.close() 
    os.remove('Data_script4.csv')

How can I resolve this issue ?

megha
  • 621
  • 2
  • 11
  • 36

1 Answers1

1

Your question was answered here

Here's the takeaway:

You need to substitute:

df = pd.read_csv("Data_script4.csv")

with this:

df = pd.read_csv('Data_script4.csv', error_bad_lines=False)
Rami Ma
  • 966
  • 1
  • 7
  • 13
  • this got rid off the above error but throws this new error - skipping line 53: expected 9 fields saw 10 fields, skipping line 100: expected 9 fields saw 10 fields..and so on. In final csv file it shows less count. I was able to get all data with above script. How can I get all data with no errors ? – megha Oct 23 '20 at 12:56