9

The title says it all. Is there any way to serialize a function generated by sympy.lambdify?:

import sympy as sym
import pickle
import dill
a, b = sym.symbols("a, b")
expr = sym.sin(a) + sym.cos(b)
lambdified_expr = sym.lambdify((a, b), expr, modules="numpy")
pickle.dumps(lambdified_expr) # won't work
dill.dumps(lambdified_expr) # won't work either

... The reason I want to do this is because my code generates so many lambdified functions but I found it takes too long every time.

denfromufa
  • 5,610
  • 13
  • 81
  • 138
akai
  • 2,498
  • 4
  • 24
  • 46

2 Answers2

10

You actually can use dill to pickle it. The most recent versions of dill (e.g. on github) has "settings" that allow variants of how the pickle is constructed on dump. Yes, the default settings for dill fail on this object, but not if you use the setting that recursively traces global references (i.e. recurse = True). This setting is similar to what cloudpickle gives you by default.

>>> import sympy as sym
>>> import pickle
>>> import dill
>>> a, b = symbols("a, b")
>>> a, b = sym.symbols("a, b")
>>> expr = sym.sin(a) + sym.cos(b)
>>> lambdified_expr = sym.lambdify((a, b), expr, modules="numpy")
>>> 
>>> dill.settings
{'recurse': False, 'byref': False, 'protocol': 2, 'fmode': 0}
>>> dill.settings['recurse'] = True
>>> dill.dumps(lambdified_expr)
'\x80\x02cdill.dill\n_create_function\nq\x00(cdill.dill\n_unmarshal\nq\x01U\x83c\x02\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00C \x00\x00s\x14\x00\x00\x00t\x00\x00|\x00\x00\x83\x01\x00t\x01\x00|\x01\x00\x83\x01\x00\x17S(\x01\x00\x00\x00N(\x02\x00\x00\x00t\x03\x00\x00\x00sint\x03\x00\x00\x00cos(\x02\x00\x00\x00t\x01\x00\x00\x00at\x01\x00\x00\x00b(\x00\x00\x00\x00(\x00\x00\x00\x00s\x08\x00\x00\x00<string>t\x08\x00\x00\x00<lambda>\x01\x00\x00\x00s\x00\x00\x00\x00q\x02\x85q\x03Rq\x04}q\x05(U\x03cosq\x06cnumpy.core.umath\ncos\nq\x07U\x03sinq\x08cnumpy.core.umath\nsin\nq\tuU\x08<lambda>q\nNN}q\x0btq\x0cRq\r.'

P.S. I'm the dill author, so I'd know.

user3666197
  • 1
  • 6
  • 50
  • 92
Mike McKerns
  • 33,715
  • 8
  • 119
  • 139
  • why recurse=True is not a default option? – denfromufa Jul 15 '15 at 16:37
  • 3
    Because `dill` can serialize a broad range of objects, and `recurse=False` provides the broadest range of coverage. If you use `recurse=True`, it works really well for certain objects, but breaks serialization for a number of cases. If (or when) I resolve the cases it breaks, then it will become the default. Until then, the best choice is to *not* remove default functionality, but provide new functionality via a setting. See: https://github.com/uqfoundation/dill/issues/105 – Mike McKerns Jul 15 '15 at 18:54
  • @MikeMcKerns I have one caveat, for reasonably large lambadified functions I get `RuntimeError: maximum recursion depth exceeded while calling a Python object`. – Josh Albert Dec 15 '16 at 17:21
  • @JoshAlbert: I expect you are seeing a specific case of pickling with lambdas. `dill` should pickle the lambda, regardless of the size of the function. If you can demonstrate a specific case of failure, please submit a question on SO, or a ticket on dill's github page. – Mike McKerns Dec 19 '16 at 14:31
  • 1
    Apparently, `dill.dump` has a `recurse` argument: `Signature: dill.dump(obj, file, protocol=None, byref=None, fmode=None, recurse=None)`. Is this the same option you can set globally in the snippet you provided? So, instead of using `dill.settings['recurse'] = True` could I just use `dill.dump(..., recurse=True)`? – Spiros Jan 27 '17 at 16:49
  • 2
    @Spiros: That's correct, you can set it on a per-call basis. – Mike McKerns Jan 27 '17 at 16:57
  • @MikeMcKerns: thanks. When deserializing a rather complex expression I previously dumped with `dill`, I got this error message: `TypeError: argument is not an mpz`. Is this something known? Dill works with simple expressions for me, but complex expressions trigger this problem. – Spiros Jan 27 '17 at 17:07
  • @Spiros: That is not a `dill` error. You can turn on `dill.detect.trace(True)` and see where it comes from. I'd google a bit to see which package throws that error. – Mike McKerns Jan 27 '17 at 19:35
4

Indeed - pickle, cPickle, and even dill fail on this example with default settings.

But cloudpickle does not fail!

pip install cloudpickle

or

https://github.com/cloudpipe/cloudpickle

import sympy as sym
from cloudpickle import dumps, loads
a, b = sym.symbols("a, b")
expr = sym.sin(a) + sym.cos(b)
lambdified_expr = sym.lambdify((a, b), expr, modules="numpy")
var=dumps(lambdified_expr)
a1=lambdified_expr(10,10)
del lambdified_expr
lambdified_expr=loads(var)
a2=lambdified_expr(10,10)
a1==a2  # True
denfromufa
  • 5,610
  • 13
  • 81
  • 138