3

I have a dictionary that I am writing to a file and then loading it again. The dictionary has some sets, so I have the option of either doing an eval() or json.dumps() with a custom ModelEncoder.

I tried both and json has a much smaller memory footprint than eval does. Why does eval take up so much more memory?

David542
  • 104,438
  • 178
  • 489
  • 842
  • `eval` is a much more general operation. Why is it surprising that it isn't as well-optimized as a tightly-constrained one? (Also, given the extent to which those constraints rule out several classes of security bugs, why are you even *considering* `eval`?) – Charles Duffy Jan 30 '17 at 23:33
  • @CharlesDuffy it's my data that i'm saving and reading so it's a quick way to do it. – David542 Jan 30 '17 at 23:35
  • Trusting your data is, generally speaking, an error. A well-designed deployment model will generally have code be read-only, but can't avoid having data be read-write; by evaluating data as code, you're thus defeating your ops team's efforts to prevent an attacker who can store arbitrary data from being able to execute arbitrary code. – Charles Duffy Jan 30 '17 at 23:37
  • 1
    This is **absolutely** not what eval should be used for. Consider using [`pickle`](https://docs.python.org/3/library/pickle.html), [`msgpack`](http://msgpack.org/index.html) or similar library. – wim Jan 30 '17 at 23:40
  • Also, `eval` is buggy when used in this context. See [Is JSON syntax a strict subset of Python syntax?](http://stackoverflow.com/questions/6627635/is-json-syntax-a-strict-subset-of-python-syntax) – Charles Duffy Jan 30 '17 at 23:40
  • ++[`msgpack`](http://msgpack.org/index.html). – Charles Duffy Jan 30 '17 at 23:44

1 Answers1

8

Whether or not you should be using eval is really another question.

The primary reason eval takes more memory is the text is first compiled into a python expression (parsed, converted to byte code) and then evaluated.

Especially with large literals, python has a memory leak in the bytecode compiler. This issue talks about pyc compilation, but you'll also hit it with code compilation in eval.

json doesn't suffer from this because it's not converting your text into executable bytecode.

anthony sottile
  • 61,815
  • 15
  • 148
  • 207