1

In my Python code I use a third party shared object, a .so file, which I suspect to contains a memory leak.

During the run of my program I have a loop where I repeatedly call functions of the shared object. While the programm is running I can see in htop, that the memory usage is steadily increasing. When the RAM is full, the programm crashes with the terminal output killed. My assumption is, that if the memory leak is produced by the shared object, because otherwise Python would raise an Exception.MemoryError.

I tried using reload(modul_name) followed by a gc.collect() but it did not free the memory according to htop.

What shall I do?

Community
  • 1
  • 1
Framester
  • 33,341
  • 51
  • 130
  • 192

2 Answers2

1

Check whether there are any object destruction/deallocation functions provided by the library so that clients can cause the memory to be freed.

See if you can identify whether it is a single function that causes the apparent leak, and then try to avoid calling that function.

Other than that, unless you have the shared library's source code or the maintainer's ear, there's not much that you can do.

mhawke
  • 84,695
  • 9
  • 117
  • 138
  • Thanks for your expertise. Is there another way to force python to drop and reload a so library or is this not possible, or just not working it that case here? I will try to get a hold of the source code. – Framester Aug 11 '15 at 12:59
  • And what about the other assumptions. Do you think they are correct? – Framester Aug 11 '15 at 13:11
1

As a temporary fix you can use a decorator to run your function in a sperate process, but eventually you will need to fix the underlying leak.

import multiprocessing
def memory_leak(func):
  def conn_func(conn, *args, **kwargs):
    conn.send(func(*args, **kwargs))
    conn.close()

  def new_function(*args, **kwargs):
    parent_conn, child_conn = multiprocessing.Pipe()
    p = multiprocessing.Process(target=conn_func, args=[child_conn]+list(args), kwargs=kwargs)
    p.start()
    result = parent_conn.recv()
    p.join()
    return result

  return new_function
Max Bazik
  • 11
  • 3