clojure application leaks memory using eval

Question

My clojure application evaluates code defined in separate .edn files at runtime, i.e. if the .edn files are changed, the contained function definitions are reloaded into an atom, which is constantly used for calculations.

The application seems to fill up the JVM's metaspace (without bounds) after a while if it's not limited. Heap space usage is normal.

I used the "YourKit" profiler to track down the leaks. It seems that most allocations, including those where the profiler says "unreachable from Garbage Collector roots", come from the eval call:

(defn eval-edn [e params_ dynparams_]
  (let [input e
        pairs (seq input)]
    (binding [*ns* (find-ns 'myapp.core)
              moods (:moods input)
              last-moods @state/moods-atom
              effects (:effects input)
              last-effects @state/effects-atom
              params params_
              dynparams dynparams_
              param-stats @state/coreparam-stats]
      (eval input))))

Where params and dynparams are dynamic vars.

The .edn file looks like this:

{:moods {:happiness 
          (* (:happiness:factor dynparams)
             (* 0.5 (Math/sin (* (/ (- (:weather:temperature params 0) 10) 30) 
                                 Math/PI))))
; ...
}}

Specifically, the memory snapshot will show thousands of objects of type myapp.core$evalN.invoke(), where N is some index increased for every new call.

I've read that eval should be avoided where possible, but I don't see how I can achieve reloading code at runtime in a JAR executable without it.

How can I make sure all the memory used by eval is cleaned up correctly?

Without addressing the memory leak itself, instead of storing function definitions in a custom EDN structure, why not just write them as as regular function definitions in a .clj file, and use [load-file](http://clojuredocs.org/clojure.core/load-file)? — bsvingen, Apr 20 '15 at 12:57
I experience the same issue. My app sets up a secure tester (with clojail) which ultimately evals a user submitted script. Loaded classes grow on each eval, and as consequence the metaspace fills up over time. — Cesar, Jun 25 '15 at 02:36

score 0 · Answer 1 · answered May 09 '15 at 14:14

0

You shouldn't be using eval. It isn't intended for your use case. Use tools.reader instead.

https://github.com/clojure/tools.reader

Detailed discussion here:

http://www.learningclojure.com/2013/02/clojures-reader-is-unsafe.html?m=1

answered May 09 '15 at 14:14

noahlz

10,202
7
56
75

Sorry I'm reviving this. But here's the thing, I'm now coming back to the code for another project (the old one I just left using eval and setting JVM options to keep leaks in check); and I still haven't understood your point. How do I use `tools.reader` to cause recursive evaluation? I tried using `#=` in the string (as in the linked example) to cause evaluation, but this doesn't seem to respect bindings of dynamic vars for the call to `read-string`... – pholz Jan 05 '16 at 09:43

score 0 · Answer 2 · answered Feb 02 '16 at 10:32

I would suggest that you keep "data" separate from "execution" (eval). Presumably what you change in your EDN-file (aka data) are keywords and numbers.

So what if you in stead keep the operations defined in your main code in stead of in your data-file, and let data be data. If your data also need to specify what operations to do, you can indicate that with keywords also, and then let your executing code act accordingly.

Keeping data and executable separate seems to me both cleaner and safer, including allowing you to use the dedicated EDN-reader for reading and parsing the input-data.

clojure application leaks memory using eval

2 Answers2

Linked