2

I have a memory leak in Java in which I have 9600 ImapClients in my heap dump and only 7800 MonitoringTasks. This is a problem since every ImapClient should be owned by a MonitoringTask, so those extra 1800 ImapClients are leaked.

One problem is I can't isolate them in the heap dump and see what's keeping them alive. So far I've only been able to pinpoint them by using external evidence to guess at which ImapClients are dangling. I'm learning OQL which I believe can solve this but it's coming slowly, and it'll take a while before I can understand how to perform something recursive like this in a new query language.

Determining a leak exists is difficult, so here is my full situation:

  • this process was spewing OOMEs a week ago. I thought I fixed it and I'm trying to verify whether my fixed worked without waiting another full week to see if it spews OOMEs again.
  • This task creates 7000-9000 ImapClients on start then under normal operation connects and disconnects very few of them.
  • I checked another process running older pre-OOME code, and it showed numbers of 9000/9100 instead of 7800/9600. I do not know why old code will be different from new code but this is evidence of a leak.

The point of this question is so I can determine if there is a leak. There is a business rule that every ImapClient should be a referee of a MonitoringTask. If this query I am asking about comes up empty, there is not a leak. If it comes up with objects, together with this business rule, it is not only evidence of a leak but conclusive proof of one.

djechlin
  • 59,258
  • 35
  • 162
  • 290
  • how many concurrent ImapClient alive? can you limit the number of ImapClient object and reuse those ? – java seeker Feb 10 '14 at 17:51
  • @javaseeker I definitely cannot do that. They need to disappear and be cleaned up. – djechlin Feb 10 '14 at 17:55
  • 1
    Are you sure that ImapClient and MonitoringTask are allocated in the same class of heap, so they will be collected together? Where else do you store pointers to ImapClients other than in MonitoringTasks -- if they truly are "leaked" (in this slightly warped sense) it's because they are referenced from other objects (perhaps, eg, other ImapClients). – Hot Licks Feb 10 '14 at 18:22
  • @HotLicks ImapClient is more ephemeral than MonitoringTask so they might not be in the same class of heap, but there should always be a MonitoringTask referring to an ImapClient. I need this query to see what other objects refer to an ImapClient. Still trying to learn VisualVM/OQL. – djechlin Feb 10 '14 at 18:51
  • Your *business rule* is not going to be enforceable at the JVM level, because you can't force things out of memory manually, it just doesn't work that way. –  Feb 10 '14 at 18:58
  • 1
    Do you EVER store a reference to an ImapClient in any other object besides a MonitoringTask? And you say that ImapClient is "more ephemeral" -- how does that happen? Is the ImapClient reference set to `null`? Do you repeatedly allocate ImapClients for a single MonitoringTask? – Hot Licks Feb 10 '14 at 19:36
  • @HotLicks yes, in lots and lots of closures. And yes, ImapClients are disposable whereas MonitoringTask persists. – djechlin Feb 10 '14 at 19:54
  • 1
    If you're creating ImapClients at a much higher rate it's likely that many more will be "dead but not collected" at any point in time. – Hot Licks Feb 10 '14 at 19:56

3 Answers3

3

Your expectations are incorrect, there is no actual evidence of any leaks occuring

The Garbage Collector's goal is to free space when it is needed and only then, anything else is a waste of resources. There is absolutely no benefit in attempting to keep as much free space as possible available all the time and only down sides.

Just because something is a candidate for garbage collection doesn't mean it will ever actually be collected, and there is no way to force garbage collection either.

I don't see any mention of OutOfMemoryError anywhere.

What you are concerned about you can't control, not directly anyway

What you should focus on is what in in your control, which is making sure you don't hold on to references longer than you need to, and that you are not duplicating things unnecessarily. The garbage collection routines in Java are highly optimized, and if you learn how their algorithms work, you can make sure your program behaves in the optimal way for those algorithms to work.

Java Heap Memory isn't like manually managed memory in other languages, those rules don't apply

What are considered memory leaks in other languages aren't the same thing/root cause as in Java with its garbage collection system.

Most likely in Java memory isn't consumed by one single uber-object that is leaking ( dangling reference in other environments ).

Intermediate objects may be held around longer than expected by the garbage collector because of the scope they are in and lots of other things that can vary at run time.

EXAMPLE: the garbage collector may decide that there are candidates, but because it considers that there is plenty of memory still to be had that it might be too expensive time wise to flush them out at that point in time, and it will wait until memory pressure gets higher.

The garbage collector is really good now, but it isn't magic, if you are doing degenerate things, it will cause it to not work optimally. There is lots of documentation on the internet about the garbage collector settings for all the versions of the JVMs.

These un-referenced objects may just have not reached the time that the garbage collector thinks it needs them to for them to be expunged from memory, or there could be references to them held by some other object ( List ) for example that you don't realize still points to that object. This is what is most commonly referred to as a leak in Java, which is a reference leak more specifically.

I don't see any mention of OutOfMemoryError

You probably don't have a problem in your code, the garbage collection system just might not be getting put under enough pressure to kick in and deallocate objects that you think it should be cleaning up. What you think is a problem probably isn't, not unless your program is crashing with OutOfMemoryError. This isn't C, C++, Objective-C, or any other manual memory management language / runtime. You don't get to decide what is in memory or not at the detail level you are expecting you should be able to.

Community
  • 1
  • 1
  • I have a process running old code that shows numbers of 9000/9100 instead of 7800/9600. That's further evidence for a leak. This process also creates 9000 on startup then cycles through them relatively slowly over several days. It looks more like a leak to me, although one that could take a week or two to OOME, so I have a need for evidence pre-OOME of a leak. – djechlin Feb 10 '14 at 18:37
  • it isn't a leak unless you get an `OutOfMemoryError`, plain and simple, the GC doesn't need to collect them, and it is correct. Read my post again and if you aren't doing anything degenerate you are done, otherwise read it again until you realize what is going on. –  Feb 10 '14 at 18:38
  • That's the point of this question. I'm trying to determine why they are not being garbage collected. Perhaps I should also mention that as of last week I was having OOMEs spewing and made a code change I thought would fix it. I'm trying to determine if my fix worked without giving the process another 2 weeks to throw another OOME. – djechlin Feb 10 '14 at 18:39
  • Because the GC doesn't need to collect them, plain and simple, and there is little to nothing that you can or need to do about it. –  Feb 10 '14 at 18:40
  • Updated my question. It's false that OOME is the only way to gain evidence of a leak. – djechlin Feb 10 '14 at 18:50
  • Your quest is a fruitless one with the way the multi-generational GC works now, unless you are getting a crashing program you don't have any reason to even suspect you have any leaks, much less be able to prove it. You can't control what you want to, that just isn't how the JVM and GC work. –  Feb 10 '14 at 18:56
  • So I should assume that my attempt at fixing the leak that was causing OOMEs worked and I have no reason to believe there might be a leak if it doesn't crash again? – djechlin Feb 10 '14 at 18:58
  • That is what UnitTests are for ... write a UnitTest that would fail in lower memory and test it. –  Feb 10 '14 at 19:00
  • My unit tests didn't throw any OOMEs so don't give any evidence of a leak. – djechlin Feb 10 '14 at 19:02
  • Okay, that makes a little more sense and I certainly should consider how to do that in the future. But at present I have 20000 lines of code with a possible leak somewhere and wouldn't even know where to begin writing unit tests to cover this situation. Which is why I need other ways to determine what's keeping my potentially dangling ImapClietns alive. – djechlin Feb 10 '14 at 19:04
  • Accepting this... basically, your answer is completely correct in my situation and should be the first go-to if you have my problem, but would be good to find an answer to the more specific question as well. – djechlin Mar 22 '14 at 03:03
0

Check your code for finalizers, especially anything relating to IMapclient.

It could be that your MonitoringTasks are being easily collected whereas your IMapclient's are finalized, and therefore stay on the heap (though dead) until the finalizer thread runs.

Chaffers
  • 176
  • 9
-1

The obvious answer is to add a WeakHashMap<X, Object> (and Y) to your code -- one tracking all instances of X and another tracking all instances of Y (make them static members of the class and insert every object into the map in the constructor with a null 'value'). Then you can at any time iterate over these maps to find all live instances of X and Y and see which Xs are not referenced by Ys. You might want to trigger a full GC first, to ignore objects that are dead and not yet collected.

Chris Dodd
  • 119,907
  • 13
  • 134
  • 226
  • `WeakHashMap` has weak keys, not weak values, so this won't do what you want it to do. You could use Apache's `WeakValueHashMap` or Guava's `MapMaker().weakKeys().makeMap()` to get a map with weak values. – Laurence Gonsalves Feb 10 '14 at 17:36
  • Even easier then -- make the X/Y the key and ignore the value. I got confuesd by java's weird backwards nameing. – Chris Dodd Feb 10 '14 at 18:03
  • It's certainly imperfect, since you can't exactly lock the whole process waiting for your `gc` to take place. – djechlin Feb 10 '14 at 18:32
  • you also can't *trigger a full GC* that isn't possible, it is only a hint that is ignored in most cases –  Feb 10 '14 at 18:35
  • @JarrodRoberson: `System.gc()` will do it on most VMs, and on those where it won't there's usually a way to do it (May need to call `System.gc()` and then suspend all user threads for some time, in the worst case). – Chris Dodd Feb 10 '14 at 23:52
  • [No you can't force it](http://stackoverflow.com/questions/66540/system-gc-in-java), you can *suggest* that it do it but that is all. Issuing `System.gc()` is an anti-pattern, and will not help the OP in any fashion. Suggesting that you do this is bad advice in almost every case, and in this one a waste of time. –  Feb 11 '14 at 00:48