3

I am running a python script using pandas.read_csv to import a csv file. Pandas provides console warnings when it doesn't see what it expects such as:

Skipping line 163: Expected 41 fields in line 163, saw 42

How can I log this to a text file?

If I run the script from a command line, python > logfile.txt only the output of print shows up in the file, not the warnings.

Arkistarvh Kltzuonstev
  • 6,824
  • 7
  • 26
  • 56
Michael C
  • 195
  • 6

3 Answers3

5

Use stderr

In bash, there are 3 default streams:

STDOUT: Redirected by > or 1>, this is standard output from a program.

STDERR: Redirected by 2>, this is diagnostic output from a program.

STDIN: Input from console, use < to input it. 


./prog 2> errorlog.txt >logfile.txt

If you want to redirect ALL output to STDOUT, use:

./prog 2>&1

Here is some more information: I/O Redirection

blackbrandt
  • 2,010
  • 1
  • 15
  • 32
0

This line worked for me

python script.py >>script.out 2>&1

which appends STDOUT (1) to script.out file and redirects STDERR (2) to the same file (&1)

Note that output is appended, so you may want to call > script.out to reset the file beforehand, as in

> script.out; python script.py >> script.out 2>&1

Sources

Franco Marchesoni
  • 15
  • 7
0
import sys
sys.stderr=openfile("file.txt",'w')
df=pandas.read_csv(filepath,on_bad_lines='warn')
sys.stderr.close()

This code will store all the warnings that will be generated by pandas.read_csv

rochester
  • 311
  • 2
  • 3