How to create converters for pandas?

Question

I want to read in a CSV file with pandas. Two columns are to be multiplied by a different constant each.

params = {
    "table-columns": ['A', 'B'],
    "multiplicands": ['0.001', '0.000001']
}

converters_ = {}
for i in range(0,len(params['table-columns'])):
    column = params['table-columns'][i]
    multiplicand = params['multiplicands'][i]
    display(column + '/' + multiplicand)
    func = lambda x: float(x) * float(multiplicand)
    converters_[column] = func
display(converters_['A'](1)) # should be 1e-03
display(converters_['B'](1)) # should be 1e-06

#df = pd.read_csv('data.csv', converters=converters_)

This is the output:

'A/0.001'
'B/0.000001'
1e-06
1e-06

Column A should be multiplied by 0.001, but it is multiplied by 0.000001. Where is the bug here?

Could you please clarify why you are using lambda function inside the loop and not just directly multiplying x and multiplicand? — medium-dimensional, May 23 '22 at 22:17
Please check these posts for more details on lambda function inside a loop: https://stackoverflow.com/questions/2295290/what-do-lambda-function-closures-capture and https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result — medium-dimensional, May 23 '22 at 22:21

score 2 · Accepted Answer · answered May 23 '22 at 22:19

The issue is that func, which you've stored in converters_[0] and converters_[1], does not evaluate multiplicand until you actually call func, at which point multiplicand is "0.000001".

You can try replacing the assignment to func with this call to eval() which hardcodes the value of multiplicand into the body of the lambda:

    func = eval(f"lambda x: float(x) * {float(multiplicand)}")

Output:

A/0.001
B/0.000001
0.001
1e-06

How to create converters for pandas?

1 Answers1