11

I've created a function using some python functionality via the reticulate package, specifically opening an image using PIL:

image <- "~/Desktop/image.jpg"
pil.image <- reticulate::import( "PIL.Image", convert = FALSE )
img <- pil.image$open( image )

I then do a few things with the image (I'm extracting several crops), which works fine. Here's an example of what I'm doing (outputs is a data frame of crops I need, so crop.grid is just a vector of 4 numbers.

crop.grid <- c( outputs$x.start[x],
                outputs$y.start[x],
                outputs$x.stop[x],
                outputs$y.stop[x] )
crop.grid <- as.integer( crop.grid )
crop.grid <- reticulate::r_to_py( crop.grid )
output.array <- img$crop( box = crop.grid )
output.array$save( output.filename )

After this, I want to clear the image from memory (the images I'm opening are very large, so take a lot of memory). I try closing the image in python:

img$close()

As well as in R:

rm( img )
gc()

And replacing the object with something I know to be very small.

img <- reticulate::r_to_py( 1L )

All of these things run fine, but my RAM still registers as being very full. I try them with each of the python objects I've created, but the only thing that clears the RAM effectively is restarting the R session.

I know that within python I'd be better off opening the image using with in order to clear it at the end of the process, but I'm not sure how to implement that using reticulate.

--- UPDATE with an analogous python version:

If I do the above in python directly:

from PIL import Image
img = Image.open( "/home/user/Desktop/image.jpg" )
output = img.crop( [0,0,100,100] )

And then close things:

output.close()
img.close()

The memory clears. The same thing doesn't work from within R though. ie:

output.array$close()
img$close()
gc() # for good measure

Doesn't clear the memory.

rosscova
  • 5,430
  • 1
  • 22
  • 35
  • Have you tried wrapping it all into [with](https://rstudio.github.io/reticulate/articles/introduction.html#with-contexts) - sth like `with(pil.image$open( image ) %as% img, { img$crop(r_to_py( c(0,0,100,100)))$save("myfile.jpg") })`? – lukeA Jun 06 '17 at 00:57
  • @lukeA I hadn't, but I have now. It runs fine, but still the same problem with the RAM being full. Interestingly it's the `output.array<-img$crop(box=crop.grid)` step that loads up the memory, but I run that step repeatedly in my function, and each iteration doesn't seem to re-read the image (only the first iteration takes any significant time). closing `output.array`, `rm`, and `gc` still seem to do nothing to clear RAM. – rosscova Jun 06 '17 at 01:32
  • Maybe wrap it into with, too? E.g. `with(pil.image$open( image ) %as% img, { with(img$crop(r_to_py( c(0,0,100,100))) %as% output.array, { output.array$save("myfile.jpg") }) })` - just trial&error. – lukeA Jun 06 '17 at 01:41
  • Fair call, but still no dice. I think `with` is operating in `R` here, so only the R object is removed. The python object (I assume sitting in an environment in the background) still remains. – rosscova Jun 06 '17 at 02:24
  • This is not specific to the `reticulate` package. Chances are your objects are indeed deleted. But the OS gave memory to R and even though that memory is now internally freed by R, R hasn't released it / given it back to the OS. There are lots of posts about this. The easiest solution is to close the R session. If you do not want to do that, I often see that you can call `gc` to free the memory but in my experience this is not a reliable way to get R to give back the RAM... – asachet Jun 14 '17 at 13:58
  • See https://stackoverflow.com/questions/11579765/how-to-clean-up-memory-not-need-to-restart-my-pc and https://stackoverflow.com/questions/8813753/what-is-the-difference-between-gc-and-rm – asachet Jun 14 '17 at 14:00
  • @antoine-sac thanks for the response. You'll see in my question that I did try using `gc()` in `R` to no avail. Unfortunately, since (I assume) `reticulate` initiates an external `python` session, that session may be where the memory allocation issue lies, so running `gc()` in the `R` session makes no difference. Restarting `R` clearly closes the `python` session properly, so a `reticulate` function to do that without restarting `R` would be enough, but something to properly clear the object without closing that session would be even better. – rosscova Jun 16 '17 at 02:16

1 Answers1

4

You need to do 3 things:

  1. Explicitly create the object in Python:

    py_env <- py_run_string(
    paste(
        "from PIL import Image",
        "img = Image.open('~/Desktop/image.jpg')",
        sep = "\n"
    ),
    convert = FALSE
)
img <- py_env$img

  2. When you're done with the image, first delete the Python object.

    py_run_string("del img")

  3. Then run the Python garbage collector.

    py_gc <- import("gc")
py_gc$collect()

Steps 2 and 3 are the important ones. Step 1 is just so you have a name to delete. If there's a way to delete "implicit" Python objects (can't think of a better term), that would save some boilerplate.

Nathan Werth
  • 5,093
  • 18
  • 25
  • Hi Nathan, thanks very much for the response. As I said, I was hoping to do this with `reticulate`'s more R-side functionality, rather than using `py_run_string`. I can certainly do this with `py_run_string`, and since there have been no answers here suggesting a direct way around the problem, I think that's probably the way I need to go. I'm going to award you the bounty, since you have indeed provided a solution, even though it's not the one I was hoping for. Just a note though, I've found that running eg `img.close()` in python seems to trigger a garbage cleanup, without an explicit call. – rosscova Jun 16 '17 at 02:11