Cannot pickle lambda function in python 3

Question

Using dill pickling of lambda functions work fine in Python 2, but not in Python 3, is there any alternative?

Python 3:

import dill 
import pickle
pickle.dumps(lambda x: x**2)

pickle.PicklingError: Can't pickle at 0x104e97840>: attribute lookup on main failed

Python 2.7:

import dill
import pickle
pickle.dumps(lambda x: x**2)

cdill.dill\n_create_function\np0\n(cdill.dill\n_load_type\np1\n(S'CodeType'\np2\ntp3\nRp4\n(I1\nI1\nI2\nI67\nS'|\x00\x00d\x01\x00\x13S'\np5\n(NI2\ntp6\n(t(S'x'\np7\ntp8\nS''\np9\nS''\np10\nI1\nS''\np11\n(t(ttp12\nRp13\nc__main__\n__dict__\ng10\nNN(dp14\ntp15\nRp16\n.

score 16 · Answer 1 · answered Sep 14 '17 at 01:43

I'm the dill author.

You can use dill in python3 without using dill directly... however, it's not as nice as in python2 just yet.

>>> import dill
>>> import pickle
>>> pickle._dumps(lambda x:x*x)
b'\x80\x03cdill.dill\n_create_function\nq\x00(cdill.dill\n_load_type\nq\x01X\x08\x00\x00\x00CodeTypeq\x02\x85q\x03Rq\x04(K\x01K\x00K\x01K\x02KCC\x08|\x00\x00|\x00\x00\x14Sq\x05N\x85q\x06)X\x01\x00\x00\x00xq\x07\x85q\x08X\x07\x00\x00\x00<stdin>q\tX\x08\x00\x00\x00<lambda>q\nK\x01C\x00q\x0b))tq\x0cRq\rc__main__\n__dict__\nh\nNN}q\x0etq\x0fRq\x10.'

You'll note the _dumps. Maybe you think that's weird. It is. It's because in python3, pickle has been merged with module that used to be called cPickle. Oddly:

import _pickle gives you the old cPickle module
import pickle gives you the pickle module, with cPickle merged in
pickle.dumps is just _pickle.dumps (i.e. it's cPickle)
pickle._dumps is the old pickle.dumps function

Confusing? The crux of it is: pickle.dumps is coded in C. The 2.x version of pickle.dumps was coded in python, but now it's been replaced by what was cPickle.dumps. If you want to get to the "old" version, you can... it's pickle._dumps.

When you import dill, dill automatically registers all objects it knows how to serialize to the the pickle serialization table -- the python one, not the C one. So, in python3, that means the one connected to pickle._dumps.

I suggest using dill.dumps directly instead.

>>> dill.dumps(lambda x:x*x)
b'\x80\x03cdill.dill\n_create_function\nq\x00(cdill.dill\n_load_type\nq\x01X\x08\x00\x00\x00CodeTypeq\x02\x85q\x03Rq\x04(K\x01K\x00K\x01K\x02KCC\x08|\x00\x00|\x00\x00\x14Sq\x05N\x85q\x06)X\x01\x00\x00\x00xq\x07\x85q\x08X\x07\x00\x00\x00<stdin>q\tX\x08\x00\x00\x00<lambda>q\nK\x01C\x00q\x0b))tq\x0cRq\rc__builtin__\n__main__\nh\nNN}q\x0etq\x0fRq\x10.'

I'd like to try to get the C table to work eventually...

I can't import dill. I'm using python 3.6 and I can import pickle but when I import dill I get module not found error — fred russell, Feb 18 '18 at 04:11
actually i spoke to soon. i pip installed dill but i'm still getting a module not found error. — fred russell, Feb 18 '18 at 04:18
check to see that your `pip` is pointing to the same `python` you are trying to use. — Mike McKerns, Feb 20 '18 at 22:46

Håken Lid · Accepted Answer · 2017-09-13T15:30:45.657

Seems like in python 2, dill replaces pickle when you import. In python 3, you have to use dill directly instead.

This works in python 3.5:

>>> import dill 
>>> dill.dumps(lambda x: x**2)
b'\x80\x03cdill.dill\n_create_function\nq\x00(cdill.dill\n_load_type\nq\x01X\x08\x00\x00\x00CodeTypeq\x02\x85q\x03Rq\x04(K\x01K\x00K\x01K\x02KCC\x08|\x00\x00d\x01\x00\x13Sq\x05NK\x02\x86q\x06)X\x01\x00\x00\x00xq\x07\x85q\x08X\x07\x00\x00\x00<stdin>q\tX\x08\x00\x00\x00<lambda>q\nK\x01C\x00q\x0b))tq\x0cRq\rc__builtin__\n__main__\nh\nNN}q\x0etq\x0fRq\x10.'

Alternatively you can also import dill as pickle

>>> import dill as pickle 
>>> pickle.dumps(lambda x: x**2)

Cannot pickle lambda function in python 3

2 Answers2