9

I have to load this massive object A (that can weight almsot 10go) that I need to pass to a function, that extracts from it a parameter B to further exert some heavy computations on it.

A = load(file)

def function(A):    
   B = transorm(A)    
   B = compute(B)
   return(B)

In order to free some memory (as I already had a MemoryError), I'd like to remove A from memory right after its transformation to B. I tried del but it doesn't seem to affect the existence of A at the script level. I also tried del global()["A"] but it says that A is not defined as a global variable.

Is there a way to do it? Thanks!

Dominique Makowski
  • 1,511
  • 1
  • 13
  • 30
  • Perhaps if you load A from inside the function() to begin with, you could take advantage of the fact that it will only 'exist' until the function returns, since it will then go out of scope... somebody correct me if I'm off here – Totem Jan 04 '17 at 10:12
  • Have you tried deleting _file_ after calling the function? – Jalo Jan 04 '17 at 10:14
  • I would really think about choosing another environment for such a task - without automatic garbage collection. Maybe it's possible to not load the whole object in first place? –  Jan 04 '17 at 10:25
  • you are most probably working with the file in a wrong way, the problem may be in the load fn and how you are handling the file and the "a" problem is just the result.. – Vitaliy Terziev Jan 04 '17 at 10:50

4 Answers4

3

del A will simply remove A from the local scope of function (see this answer). A will still persist in the global scope. To remove it from the global scope you can either use a closure (and declare global A) or with python3 you can also use the keyword nonlocal. However this only removes the binding from the scope and does not guarantee that the corresponding memory is freed. This happens when the object is garbage collected. You can force garbage collection via the gc module (see this answer).

However if you are running into memory problems, instead of loading the whole data set, you could maybe use a view onto the data set and only process (load) a part of it at a time (i.e. stream-process the data).

Community
  • 1
  • 1
a_guest
  • 34,165
  • 12
  • 64
  • 118
0

Perhaps loading the object from inside the function would work here, since A will go out of scope once the function returns and no longer take up the memory in the same way(A will probably still exist in memory, but that memory should now be available for other use again when needed). Maybe try something like this:

f = file                 # assuming file is not the memory hog

def function_A(file):
    A = load(file)       # A is created in the local scope of the function
    return transform(A)  # A will go out of scope, freeing the memory for use

def function_B(file): 
   B = function_A(file)  # when this returns the memory should be available again
   return compute(B)

Then just call function_B(file)

Totem
  • 7,189
  • 5
  • 39
  • 66
0

I believe reassigning A within the function could achieve the effect you are looking for.

def function(A):
    B = transform(A)
    A = None
    B = compute(B)
    return(B)
Kumar
  • 1
  • 6
    Setting A = None inside the function will not change the value of A outside the function after it has returned. A = None there only sets a value for A in the local scope. – Totem Jan 04 '17 at 10:25
  • 2
    Imagine people that you are at a large business convention or meeting. There are many people wearing write-on name-tags/stickers on their shirts. There are many people at this business function, including a Ms. Sarah, and "Mr. None". Initially, Sarah is wearing a name-tag on her shirt on which says "`A`". Executing the code `A = None` is like (1) taking the "`A`" name-tag off Ms. Sarah (2) sticking the "`A`" name-tag onto Mr. None. Executing the code `A = None` does ***not*** remove Sarah from the convention. Before, Sarah's name-tag said, "A." – Toothpick Anemone Sep 30 '20 at 16:09
  • Afterwards, Sarah has no name-tag at all, the `A` name-tag is moved to Mr. None's shirt . – Toothpick Anemone Sep 30 '20 at 16:09
0

declare the outside variable as global

a = 1

def func():
    global a
    print(a)
    del a

func()
print(a)
probhonjon
  • 302
  • 5
  • 8