Converting number data from CSV to a float and getting average in Python

Question

I have the following python function that takes an input of directory and the file name.

It reads from a CSV file which can have a line with 'null' as the value.

tps.csv

1
2
3
4
null
5
6

The moment it reads 'null' from the csv, it gives me an error (below) even though I have an if condition to skip when it reads 'null.'

ValueError: could not convert string to float: null

Code:

def computeMean(dir_name, filename):
    csv_filename = '{}/{}'.format(dir_name, filename)
    numbers = []
    with open(csv_filename) as f:
        for line in f:
            if line is not 'null':
                number_on_line = float(line)
                numbers.append(number_on_line)

    return sum(numbers)/len(numbers)

What do I need to add or change so that it ignores/skips non-numerical values?

Try changing if condition to - if 'null' in line instead of if line is not null — Dinesh Pundkar, Sep 16 '16 at 09:45

score 2 · Answer 1 · edited May 23 '17 at 12:00

2

In your file, a line ends with \n. You have then the string null\n, not null. The best way to handle that is probably to use strip method, which is designed for that:

if line.strip() != 'null':
    ...

Note that is is not used here. Because is doesn't work as you think.

edited May 23 '17 at 12:00

Community

1
1

answered Sep 16 '16 at 09:47

aluriak

5,559
2
26
39

Daniel Hepper · Answer 2 · 2016-09-16T09:52:52.643

1

Use a comparison instead of is not:

if line != 'null':

is/is not checks for object identity, while ==/!= compares the value of objects. In your case, the line has the same value as the constant string null, but they are different objects.

Note that due to the internals of Python, comparing strings with is will work sometimes, but not always.

>>> 'foo' is 'foo'
True
>>> 'foobar'[:3] is 'foo'
False

edited Sep 16 '16 at 09:52

answered Sep 16 '16 at 09:47

Daniel Hepper

28,981
10
72
75

Thank you for explaining this. – jeffsia Sep 16 '16 at 09:58

score 1 · Answer 3 · answered Sep 16 '16 at 09:48

1

is operates differently than str in str or str == str, thus the error

I believe you can do this with a list-comprehension

with open(csv_filename) as f:
    numbers = [float(l.strip()) for l in f if l != 'null']
return 0 if len(numbers) == 0 else sum(numbers) / float(len(numbers))

answered Sep 16 '16 at 09:48

OneCricketeer

179,855
19
132
245

Thanks. This is the shortest solution that I've found. I've combined this with @aluriak answer and added the .strip() in the list comprehension for l in f if l.strip() != 'null' – jeffsia Sep 16 '16 at 09:56

Converting number data from CSV to a float and getting average in Python

3 Answers3