GC and YGC occurrence and next action of course

Question

We have a java application which I can say utilises quite a number string variables too. In addition we started it with both max and initial memory set as 256Mb. Thereafter it ran for sometime till lately we found that first it was one GC and after nearly 2 weeks another one more GC. Thus we found the average of FC is 0.2145s and YGC is 0.0511s. We are not sure it these values are suitable or leading to something else? We decided to increase the memory to 512Mb and also looking into some other GC setting such as the CMS settings? Should also optimise on the codes such as usage of the String variables? Normally what are object or variables type will move into the Old Objects ?

Timestamp     S0      S1     E     O      P      YGC    YGCT      FGC FGCT   GCT
   2752472.6  69.74   0.00   3.67  51.72  44.17   4614  235.630   2   0.429  236.059
   2752477.6  69.74   0.00   5.38  51.72  44.17   4614  235.630   2   0.429  236.059
   2752482.6  69.74   0.00   7.51  51.72  44.17   4614  235.630   2   0.429  236.059
   2752487.6  69.74   0.00   9.52  51.72  44.17   4614  235.630   2   0.429  236.059
   2752492.6  69.74   0.00  10.55  51.72  44.17   4614  235.630   2   0.429  236.059

Code Snippet

BoneCP connectionPool = null;
  class ConnectionHandler implements Runnable {


    private Socket receivedSocketConn1;
    ConnectionHandler(Socket receivedSocketConn1) {
      this.receivedSocketConn1=receivedSocketConn1;
    }
    Connection dbconn = null;

    public void run() { // etc
     BufferedWriter writeBuffer = null;
     BufferedReader readBuffer = null;
     String capturedMessage="";

     try{
        dbconn = connectionPool.getConnection();
        dbconn.setAutoCommit(false);

        while ((nextChar=readBuffer.read()) != -1){          
          capturedMessage += (char) nextChar;

          if (nextChar == '*')
          {
           try{

                //all queries here.
                //for insert query this how i do
                Statement stmt1 = null;
                stmt1 = dbconn.createStatement();
                String insertQuery3 =........
                count = stmt9.executeUpdate(insertQuery3);
                try{
                  if ( stmt1!= null ){  stmt1.close();
                }
                else{
                System.out.println("No stm1 exist");

                }
                }catch(SQLException ex){   
                  System.out.println("SQLException has been caught for stmt1");
                  ex.printStackTrace(System.out);
                }

                ///For select we do this  

                Statement stmt2 = null;
                stmt2 = dbconn.createStatement(); 
                String selectQuery2= .........
                ResultSet rs2 = stmt2.executeQuery(selectQuery2);

                if(rs2.next())
                {

                }
                try{
                 if ( rs2!= null ){  
                     rs2.close();
                 }   else{
                 System.out.println("No rs2 exist");
                 }

                 if ( stmt2!= null ){ 
                      stmt2.close();
                 }   else{
                 System.out.println("No stm2 exist");

                 }
                }catch(SQLException ex)
                {   
                System.out.println("SQLException has been caught for stmt2");
                ex.printStackTrace(System.out);
                }



               dbconn.commit
            }
           catch (SQLException ex){
                ex.printStackTrace(System.out);
                try{  
              dbconn.rollback();
           }
           catch (Exception rollback){  
              rollback.printStackTrace(System.out);
              }
       }
       catch (Exception e){
           e.printStackTrace(System.out);
           try{  
              dbconn.rollback();
           }
           catch (Exception rollback){  
              rollback.printStackTrace(System.out);
             }
       }
       finally
     {

     }


        }
     catch (SocketTimeoutException ex){
           ex.printStackTrace();
     }
     catch (IOException ex){
           ex.printStackTrace();
     }
     catch (Exception ex){
           ex.printStackTrace(System.out);
     }    
      finally{
        try{
         if ( dbconn != null ){
           dbconn.close();
         }
         else{
          System.out.println("dbConn is null in finally close");
         }
        }
        catch(SQLException ex){
            ex.printStackTrace();
        }
        try{
          if ( writeBuffer != null ){
            writeBuffer.close();
         }
         else{
          System.out.println("w is null in finally close");
         }
        }
        catch(IOException ex){
            ex.printStackTrace(System.out);
        }
       }
      }
    }

Finalizer analysis from sun.misc.Launcher$ExtClassLoader @ 0xe003a648 (50%) Size: 11.2 MB Classes: 196 Objects: 201.7k Class Loader: 7

A total of 91 objects implement the finalize method.

 Histogram of Objects with Finalize Method
Class Name                         Objects Shallow Heap 
com.mysql.jdbc.JDBC4Connection
First 10 of 38 objects             38      42,560 
java.net.SocksSocketImpl
First 10 of 39 objects             39      4,368 
java.util.jar.JarFile
All 6 objects                      6        384 
java.util.zip.Inflater
All 6 objects                      6        192 
java.util.concurrent.ScheduledThreadPoolExecutor
All 1 objects                      1        80 
java.util.concurrent.ThreadPoolExecutor
All 1 objects                      1        72 
Total: 6 entries
 91 47,656

Finalizer analysis from

(46%) Size: 13.6 MB Classes: 578 Objects: 304.8k Class Loader: 4

A total of 19,341 objects implement the finalize method.

Class Name                        Objects Shallow Heap 
java.net.SocksSocketImpl
First 10 of 18,354 objects        18,354  2,055,648 
com.sun.mail.smtp.SMTPTransport
First 10 of 162 objects           162     25,920 
java.io.FileOutputStream
First 10 of 742 objects           742     23,744 
com.mysql.jdbc.JDBC4Connection
First 10 of 15 objects            15       16,800 
java.util.jar.JarFile
First 10 of 20 objects            20       1,280 
java.util.zip.Inflater
First 10 of 25 objects            25       800 
java.util.concurrent.ThreadPoolExecutor
All 8 objects                     8        576 
sun.net.www.protocol.jar.URLJarFile
All 5 objects                     5        400 
java.util.concurrent.ScheduledThreadPoolExecutor
All 2 objects                     2        160 
java.lang.ClassLoader$NativeLibrary
All 4 objects                     4        128 
java.io.FileInputStream
All 3 objects                     3        96 
sun.jdbc.odbc.JdbcOdbcDriver
All 1 objects                     1        24 
Total: 12 entries
 19,341 2,125,576

Finalizer analysis from org.rzo.yajsw.boot.WrapperClassLoader @ 0xe02818c0 (4%) Size: 966.3 KB Classes: 361 Objects: 19.9k Class Loader: 3

A total of 22 objects implement the finalize method.

 Histogram of Objects with Finalize Method
Class Name                              Objects Shallow Heap 
java.util.jar.JarFile
First 10 of 11 objects                  11      704 
java.util.zip.Inflater
All 10 objects                          10      320 
java.util.concurrent.ThreadPoolExecutor
All 1 objects                           1       72 
Total: 3 entries
 22 1,096

Histo Live Results.

 num     #instances         #bytes  class name
----------------------------------------------
   1:          6001       18078232  [I
   2:         44562        7196072  [B
   3:        103183        6054240  [C
   4:         31832        4786936  <constMethodKlass>
   5:         31832        4340240  <methodKlass>
   6:         31638        3543456  java.net.SocksSocketImpl
   7:         20127        3542352  com.mysql.jdbc.JDBC4ResultSet
   8:         41022        3461576  [Ljava.util.HashMap$Entry;
   9:        104185        3333920  java.lang.String
  10:         20198        3070096  com.mysql.jdbc.StatementImpl
  11:         20323        2926512  com.mysql.jdbc.Field
  12:         60329        2895792  java.util.TreeMap
  13:          2343        2800032  <constantPoolKlass>
  14:         61239        2449560  java.util.TreeMap$Entry
  15:        149690        2395040  java.lang.Object
  16:         41000        1968000  java.util.HashMap
  17:          2343        1895080  <instanceKlassKlass>
  18:          1954        1569536  <constantPoolCacheKlass>
  19:         31623        1517904  java.net.SocketInputStream

Initial Start Of Socket

71:             2            224  java.net.SocksSocketImpl
176:            1             32  java.net.Socket
202:            1             24  java.net.InetSocketAddress
206:            1             24  java.net.ServerSocket

After running 50 Sockets.

 19:            52           5824  java.net.SocksSocketImpl
 37:            50           2400  java.net.SocketInputStream
 48:            51           1632  java.net.Socket
 88:             8            384  java.net.SocketOutputStream
 95:             8            320  java.net.SocketTimeoutException
 135:            7            112  java.net.Socket$3
 136:            7            112  java.net.Socket$2
 229:            1             24  java.net.ServerSocket

score 1 · Answer 1 · answered Oct 21 '13 at 17:23

1

512 MB doesn't sounds that much and it may not be worth your time trying to optimise it further. BTW 512 Mb is 512 Mega-bit.

Medium and long term lived objects end up in old generation.

You can optimise your code with a memory profiler to reduce your String usage. You can use StringBuilder to recycle the objects and reduce GC pressure which is a common cause of objects ending up in tenured space incorrectly.

Taken to it's extreme, you might find you can eliminate all String generation, but I doubt it is worth and increasing memory to 1 GB may be simpler/cheaper.

answered Oct 21 '13 at 17:23

Peter Lawrey

525,659
79
751
1,130

how to decide what is medium and long term lived objects any examples? So you are saying dont need to change to stringbuilder is it? – user2711681 Oct 21 '13 at 17:30
1

Medium term objects live long enough to reach tenured space and then die. Long term objects last in old gen a long time. StringBuilder is an option but I can't see the need when you can just a bit more memory. I use YourKit but I would try VisualVM first as it is free and built in. – Peter Lawrey Oct 21 '13 at 17:44
1

You can buy 64 GB for less than £500, or about $12 a GB. How much effort is worth spending to save a few $ of memory (assuming that is what it costs you or your client of course). – Peter Lawrey Oct 21 '13 at 17:46
Ok I will take your advice to increase to 1GB. Your suspect is memory expansion is it? Actually I know memory tuning needs some time as I read quite a number articles on it. What I dont get is that what are the object are they my socket thread? My db connection polling? Is there a method to optimise on this or normally just increase on the heap memory? – user2711681 Oct 21 '13 at 17:47
actually is not about saving the cost but I want to make sure that there is nothing wrong in my application such as leakages etc. That is my only worry here. So what trend should I look out on the jstat? – user2711681 Oct 21 '13 at 17:50
What I want to understand first is on the objects how they created and how they survived from Eden to S1 to Old Gen? – user2711681 Oct 21 '13 at 17:59
To check for a memory leak, you need to look at the memory used after a full GC over an extended period. You are not getting much in the way of full GCs, so there isn't much evidence. – Peter Lawrey Oct 21 '13 at 18:04
A profiler can help you reduce the memory allocated and help reduce pressure on the GCs. You might like to try this as an exercise. However at some point you have to trade off your time with the memory you save (esp if this is memory which wouldn't be used otherwise) – Peter Lawrey Oct 21 '13 at 18:07
What to look and based on my full GCs what evidence suggest that is not ? Maybe you can give me some hints so I can look and monitor it closely? Because I am kind of confuse on how to decide if there is more memory needed or there is memory problem (e.g memory leak) – user2711681 Oct 21 '13 at 18:09
The trend you should look for is increasing memory *used* after a full GC and increasing occurance of full gcs. – Peter Lawrey Oct 21 '13 at 18:09
When you say increasing in memory means which one you mean to say the old generation or pem generation? Because I do notice after the full GC the old gen do keep increasing? – user2711681 Oct 21 '13 at 18:11
Even small increases can normal or ok if the application is warming up. A memory leak is one which is an undesirable increase, and what is undesirable depends on the application usage. – Peter Lawrey Oct 21 '13 at 18:13
@Actually the application is socket application where it receive string of data and there go through a number of function which involves a number select, insert and update statement. All this are using db connection pooling and every one resultset and statement are closed after used immediately. Which one is the jstat memory to be monitored for increase ? – user2711681 Oct 21 '13 at 18:15
It can be old or perm gen, but I never use so many classes that perm gen is a problem. I normally worry about old gen and ignore perm gen as a problem. If would ignore the first few old gen and only look at memory retained, not allocated. – Peter Lawrey Oct 21 '13 at 18:15
I dont think the memory retain is shown here in jstat that is why jmap right? What I notice is that after every YGC there is a small increase in my OLD Gen is that fine ? – user2711681 Oct 21 '13 at 18:17
So most of your objects should be short lived. Each young collection pushes some objects into tenured space and often those die in the next Full Collection. – Peter Lawrey Oct 21 '13 at 18:33
Sorry peter in my case what will be the object is it each of the socket connection? I am not clear what is the object here is it each instance of the socket? So can I say some socket object will go into tenured space and die during the FGC? – user2711681 Oct 21 '13 at 18:39
If the socket exists over many collections this will happen. The Socket object itself is usually self evident like SocketImpl or SocketChannelImpl. – Peter Lawrey Oct 21 '13 at 18:44
Basically this how my programme goes class ConnectionHandler implements Runnable { private Socket receivedSocketConn1; ConnectionHandler(Socket receivedSocketConn1) { this.receivedSocketConn1=receivedSocketConn1; } – user2711681 Oct 21 '13 at 18:49
Then I have public void run() {} where in it I process the socket to receive data and lastly close the socket. So in this case can I considered each of the receivedSocketConn1 as one new object instance of the whole class? – user2711681 Oct 21 '13 at 18:51
You can consider each class of your choice as a connection. However, there is usually many other classes associated with the connection/socket. – Peter Lawrey Oct 21 '13 at 18:54
Yes each connection I think as an object right. When you say many other classes accociated you mean in build or my own classes? Offcourse I got a whole lists of import statement. – user2711681 Oct 21 '13 at 19:00
import statements don't matter only the objects created with `new` by you or by libraries you call. – Peter Lawrey Oct 21 '13 at 19:02
Ok I do have this DateFormat dateFormat = new SimpleDateFormat("dd/MM/yyyy HH:mm:ss"); DateFormat formatter = new SimpleDateFormat("EEE, dd MMM yyyy"); DateFormat inDf=new SimpleDateFormat("ddMMyyHHmmss"); DateFormat outDf=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); DateFormat outDfA=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); – user2711681 Oct 21 '13 at 19:07
DateFormat dateFormat1 = new SimpleDateFormat("dd/MM/yyyy HH:mm:ss"); Date dateIn1 = new Date(); and Properties props = new Properties(); props.put("mail.smtp.host", "*******"); props.put("mail.smtp.socketFactory.port", "****"); and Message emailMessage = new MimeMessage(session); and URL url = new URL(request); and BoneCPConfig config = new BoneCPConfig(); and Socket socketConn1 = serverSocketConn.accept(); new Thread(new ConnectionHandler(socketConn1)).start(); these is where I use all the new statements – user2711681 Oct 21 '13 at 19:07
All of them create objects, and many of these create more objects you can't see. – Peter Lawrey Oct 21 '13 at 19:12
Thank you now I am getting things clearer so I guess there are occupies of the generation from EDEN to S to OLD am I right? – user2711681 Oct 21 '13 at 19:13
Yes, some objects live long enough to pass between all the generations. – Peter Lawrey Oct 21 '13 at 19:15
We cant decide which will live longer or can we say based on the type of object that will live till the old generation ? Do you think I should reduce the number of objects ? – user2711681 Oct 21 '13 at 19:17
Any type of object can die young, live for a while, or never die. Like I keep says, I doubt it worth it for your application, expect as an exercise. – Peter Lawrey Oct 21 '13 at 19:19
So your best advice just increase the memory for now and monitor the number of GC? The only thing I am lost is on the memory monitor? You said to monitor the retained where can I get that numbers from ? – user2711681 Oct 21 '13 at 19:24
@user2711681 look at the amount of old gen used after a full GC. – Peter Lawrey Oct 22 '13 at 08:26
I will look but I see it keep growing. So what pattern to be on the look out? You advice me to check the retained memory how to check that? – user2711681 Oct 22 '13 at 15:39
@user2711681 What is the size of the heap after a full GC. That is the memory retained (plus a little more depending on how long after the Full GC the sample was taken) A small increase might be normal so you need to see the behaviour over a number of full GCs. You can trigger a full GC which tools like VisualVM or jmap. – Peter Lawrey Oct 22 '13 at 15:57
Great normally I always take one or two sample of jmap on a daily basis. I do notice even without the full GC there is a drop in the heap size but then there increase as time goes.I fully depend on jmap as I have given up setting VisualVM for my remote applications. – user2711681 Oct 22 '13 at 16:06
You can `jstat` just to collection the information. `jmap` has some `live` options which triggers a full GC. – Peter Lawrey Oct 22 '13 at 16:09
normally this how I jstat ./jstat -gcutil -t 5s. But how to see the retain total memory here. For the jmap this how I run the command ./jmap -dump:format=b,file=heap231013.bin 3111 and after collecting it I normally view with MAT tool from eclipse. Can you verify if my command for jmap is correct and what best tool in your opinion to use for the viewing of retain memory? – user2711681 Oct 22 '13 at 16:30
I use `jstat`, like I said. `gcutil` is fine. I wouldn't bother with a heap dump unless you know you have a problem. – Peter Lawrey Oct 22 '13 at 16:33
How you use jstat to view the retain memory as it does not the total heap memory after gc maybe I dont know the command? All these while I have been taking jmap and then I view via MAT to view the total heap size? So what is your best solution here for me to keep monitoring on the heap size? – user2711681 Oct 22 '13 at 16:37
You can see that the total heap usage is and you can see when a GC happens. Just look at the total heap when the full gc count increases. – Peter Lawrey Oct 22 '13 at 19:12
for instance based on this jstat results how to see total heap usage Timestamp S0 S1 E O P YGC YGCT FGC FGCT GCT 2752472.6 69.74 0.00 3.67 51.72 44.17 4614 235.630 2 0.429 236.059 – user2711681 Oct 23 '13 at 10:55
@user2711681 You cannot add the percentages in this case, nor is an example of a full GC just run. After a full GC the eden and survivor spaces is empty (or as close as you will see), so you only need to look at the old gen. – Peter Lawrey Oct 24 '13 at 21:02
so when to look for it after how a long a GC run must I look the at old gen? What pattern to look for? Another thing you mention that by using jstat I can view the current heap how to do that please? – user2711681 Oct 26 '13 at 06:00
I would look immediately after. the pattern to look for is a steady increase. you can view the heap size with `jstat` but it is the size after a full gc which is interesting. – Peter Lawrey Oct 26 '13 at 11:59
is kind of difficult to know when the GC will happen rite? So what pattern to see because on a normal case after every time the eden is cleared I do notice a small increase in my old generation. How to view the heap size with jstat what inputs to add ? – user2711681 Oct 26 '13 at 15:09
You are right that is hard to know when it will happen but even if you did know this it wouldn't be very helpful. Seeing a small increase of old gen after a minor collection is completely normal. I suggest you run `jstat -usage` as there is many options which might be interesting. – Peter Lawrey Oct 27 '13 at 06:55
I am kind of confuse you mean to say run jstat -usage but that is not a valid option I guess you mean to run some other option beside gcutil like gc, gccapacity etc is it? What will be best in you experience. I found one more new command ./jmap -heap pid which is quite helpful I find. My perm is capacity is just 20.75 and 79.28% is used do you think I should increase it but my maxPermSize is 82.0M ? MY eden is shows as 150Mb but 65.93 is used which I think will be fluctuating rite. – user2711681 Oct 27 '13 at 12:51
You should increase the size of your eden space if you see to much premature promotion or it is filling in less than 2 seconds (or what ever you deem appropriate, for my applications it is 10 hours) You can use other options, but the size of the old gen after a full collection is the most useful. I don't see a problem with your perm gen. How long does it take for your eden space to fill on average at the busiest times? – Peter Lawrey Oct 27 '13 at 17:20
on average I notice the eden get filled up in less then 5 minutes what do you think should I do something about it ? On every eden clearance I notice the old is filled up by between 0.01 and 0.02. So far I just started my application there is not full GC but around 242 YGC. So should I add my eden space ? When is perm generataion and old generation is considered in danger zone? – user2711681 Oct 28 '13 at 13:50
any input on the change of the Eden space. Do I need to do anything here ? Does it need to be added? – user2711681 Oct 30 '13 at 16:16
5 minutes to fill Eden is fine, better than most applications. Small increases in old gen is fine. Old gen and perm gen are very expensive to cleanup up. That take 10 - 100 times longer which is why avoiding these collections is often critical. You might consider using the CMS collector to reduce the impact of cleaning the old gen. If the Eden space is still small compared to the main memory size I would increase it, otherwise 5 mins is good, the recommended minimum is 2 seconds. – Peter Lawrey Oct 30 '13 at 21:01
thank you for the gudie on eden. So you idea is to keep the old n perm empty which impossible. I guess you want to target for minimal increase right. So how to keep both these minimal. I have read about CMS some say its good some commented its bad. So what is your idea based on your vast experience. Another thing incase in future I need to increase eden how to do that ? – user2711681 Oct 31 '13 at 09:47
I stated that small increases in old gen are normal. CMS is good for minimising full GC pauses times esp when demand on the old gen is relatively low. – Peter Lawrey Oct 31 '13 at 10:25
ok I will enable CMS. I have using this site http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#par_gc.oom . Offcourse CMS comes with extra overheads rite. So should I just set using this UseConcMarkSweepGC or any other extra parameters needed. Off course we cant avoid the filling of old gen. So how will cms help you mean in parallel it will keep clearing old gen before it get to the maximum is it? – user2711681 Oct 31 '13 at 14:48
1

The CMS can avoid filling the old gen by cleaning it up concurrently. Ie before it fills. – Peter Lawrey Oct 31 '13 at 14:54
thank you for the confirmation. If CMS keep cleaning it will the full GC ever occur then? I guess those articles which are talking about overhead is quite negligible in terms of memory usage? What is your say in this? How about for perm generation clean up? – user2711681 Oct 31 '13 at 15:06
1

A Full GC will occur if the CMS cannot keep up with the rate you are producing garbage, i.e. if it fails to clean up enough before you fill the old gen. CMS uses much more CPU, but it is CPU you might otherwise have idle. If you have a machine where all the CPUs are busy most of the time, a parallel collector is better. Perm gen doesn't get cleaned up easily but this only occurs when you unload a class loader. Do you do that often? – Peter Lawrey Oct 31 '13 at 15:37
For now my load it not that high yet therefore most of the time all my cpu are quite in idle state and my load is always below 0.2 (I mean in my centos machine). How much CPU will CMS use and how it will be indicated be part of my java application is it? I guess in my case its CMS the solution. So far I dont get that many GC so should I implement it now ? I dont quite get what is a class loader is it linked to say each of socket object or my pooling object.When normally does it get added because I do see some small increment in my perm but I read that it should not be increasing? – user2711681 Oct 31 '13 at 18:00
1

perm gen increases as it loads new methods and compiles them. Small increases are normal as well. The CMS could add another 10% CPU, but in your cause I suspect it will be lower. If you haven't had to worry about class loaders, you don't need to worry about perm gen. – Peter Lawrey Oct 31 '13 at 18:59
as explained earlier my application is mutlithread socket listener with database pooling done. You have seen I do create new object of date etc. So does each of this thread contribute to perm generation or to the young generation. Does class loader mean loading my class each time one connection is made? Because from this link http://stackoverflow.com/questions/2424604/what-is-a-java-classloader I understand that first it loads my main function. But the whole thing is in one big class public class cS9000 { then it is the main and also class ConnectionHandler implements Runnable { – user2711681 Nov 01 '13 at 16:55
Perm gen is for loading classes for the first time and when they are compiled t native code. It wouldn't make sense to reload the class every time it is used. new objects are added to the eden space and this is where you should see most activity. – Peter Lawrey Nov 01 '13 at 19:27
@So in my case where its multi thread what is loaded is the instance of the class which is the object into the eden space. What can be causing the small increase in perm generation? Is it some thing to be monitored here? – user2711681 Nov 02 '13 at 04:47
Like I said, you can get small increases in perm gen due to loading classes you haven't loaded before or methods you haven't used yet, or optimising to native code methods which haven't been optimised yet. I have never needed to monitor the perm gen as I never used it enough to run out. I suggest you minimise how much you load classes i.e. don't load thousands and you shouldn't have a problem either. – Peter Lawrey Nov 02 '13 at 08:32
@Practically each socket will go through most of my methods thus I dont think any of the classes have not been loaded before. What is this optimising to native code effects on my perm generation? I guess this is beyond my control right? In your case does your perm generation do increase but up to what level does it increase? But in my case class public class cS9000 { then it is the main and also class ConnectionHandler implements Runnable { is each socket connection which calls this new class ConnectionHandler is consider a new class loading or its not considered? – user2711681 Nov 02 '13 at 09:10
@user2711681 The amount it increases depends on your code and your settings. Like I said, it is quite likely you will never need to worry about it. It is usually only a problem with lots of class loaders but you appear to only have one. I suggest you use `-XX:+PrintCompliation` and you can see when a method get compiled to native code, you will see this goes on long after you might expect. – Peter Lawrey Nov 02 '13 at 09:51
yes I guess my this class is just loaded once public class cS9000. I am not sure about this class ConnectionHandler implements Runnable guess is also once right unless I manually call the class loader right. I guess I know what is causing the small increase. I did a jmap and view the result via MAT and the leak suspect keep showing me this "The class "java.lang.ref.Finalizer", loaded by "", occupies 6,644,168 (42.59%) bytes. ". What is your opinion is my interpretation right about the small increase in my perm gen – user2711681 Nov 02 '13 at 09:58
jmap and jmat don't analyse the perm gen. – Peter Lawrey Nov 02 '13 at 16:04
what my worry about perm here is that its now nearly 81.35%. I worry it touches nearly full and cause some error. So that is why I want to understand it and try to catch if there any problem. What else can I do to analyse it? MY worry every few days there is a small increase I notice. – user2711681 Nov 02 '13 at 18:35
1

@user2711681 Unfortunately there isn't much in the way of tools for analysing perm gen. I would make sure it is the *maximum* perm gen you are reaching and not the current perm gen. The current size can expand as required and should get cleaned up as required anyway. – Peter Lawrey Nov 02 '13 at 18:37
@When it reach 100% does the GC clean it too ? Meaning that the GC will run when it reaches nearly 100%. So is kind of hard to tell what is causing it the very small increase I guess. What is your best advice here? – user2711681 Nov 02 '13 at 18:53
@user2711681 Unless you are loading thousands of classes, it is unlikely to be a problem and I wouldn't worry about it unless I know its a problem. – Peter Lawrey Nov 02 '13 at 18:59
I dont think so in my application is running so many classes as per my definition only 2 that is class cS9000 and ConnectionHandler implements Runnable . But what is misleading in this message "The class "java.lang.ref.Finalizer", loaded by "", occupies 6,644,168 (42.59%) bytes. ". Why is memory leak report pointing to this. I guess even the finalize have been loaded once before right. – user2711681 Nov 02 '13 at 19:02
1

The Finalizer holds references to object which have not been finalized yet. i.e. They have a `finalise()` method. A high count like this suggest you have resources you are not cleaning up. I would make sure you are not opening too many files or socket and you are always closing them off. Like I said, the heap usage gives you no idea of class loading. – Peter Lawrey Nov 02 '13 at 21:46
just to share with you yesterday my perm gen was 81.35 and today its like 81.47%. What could be leading so this very small increases? – user2711681 Nov 03 '13 at 08:19
1

Did you try `-XX:+PrintComplication` I can't think of anything more than the last three times you asked this ;) – Peter Lawrey Nov 03 '13 at 20:30
1

@Sorry for my repetition the problem I am having some issue with the -XX:+PrintComplication because I am using this tool yajsw so let me solve that then I will report here. Another thing with regards to the finalize I have ensure that I have finally for both the socket and also the db connection what else could be the problem ? – user2711681 Nov 04 '13 at 02:45
Yes, make sure you have finally blocks which close off the resources, or you can use try-with-resources block with Java 7. – Peter Lawrey Nov 04 '13 at 07:53
sorry been busy with the -XX:+PrintComplication still cant solve cause I am using the yajsw wrapper to run so wrote to the owner of this library he said just can view via console not log. I will get his help further to see if it can be done. Another thing I have update my question with snippet of my codes. IF you see for all my select, update and insert after every usage of the statement and resultset I close them. Only thing I never use here is finally cause then if there is error I cant rollback the whole thing. MY java.lang.ref.Finalizer increase from 8,015,352 yesterday to 8,657,552 today – user2711681 Nov 05 '13 at 14:54
@user2711681 It sounds like you need to know which class is creating needing these Finalizer clean up. What objects do these node point to? – Peter Lawrey Nov 05 '13 at 21:38
I have dig further into the heap analysis and found this day 1. Size: 20 MB Classes: 2.5k Objects: 414.2k Class Loader: 11 day 2. Size: 21.9 MB Classes: 2.5k Objects: 456.6k Class Loader: 11 . Further I dissect the top component and I have posted in my question all the 3 results of the finalizer. I dont know if those are helpful for me to look into. But what surprise me is that here it shows me class loader is 11. But when I sum from the top components it 14 ( 7 + 4 +3 ) and even the other like total memory does not tally with the summary. – user2711681 Nov 06 '13 at 03:20
any input from your side on this information of the finalizer and summary report. Thank you. – user2711681 Nov 07 '13 at 14:06
@user2711681 I still think you should look at what objects need to be finalized. – Peter Lawrey Nov 07 '13 at 22:00
how to look at what objects need to be finalised. I just managed to scoop all these from the MAT analysis. Any other method to retrieve on that ? Thank you. – user2711681 Nov 08 '13 at 05:05
I use yourkit/visual vm to read/take a heap dump and see which objects are consuming the most memory and why. You can browse the data and see which objects the Finalizers are pointing to and thus trying to clean up. – Peter Lawrey Nov 08 '13 at 07:48
I have actually taken heaps on a daily basis. I have tried to load via visual vm but I dont get to see many option just shows me number of classes and their instances where char[] is the highest. How to browser to see which objects the finalizer is pointing. I have actually update my question with the MAT tools results did you have a look does that make sense? E.g. A total of 91 objects implement the finalize method. etc – user2711681 Nov 09 '13 at 03:15
The JDBC4Connection can be holding on to any amount of resources. I use YourKit and it can display all the objects by size and their data structures etc, from a heap dump. It is also easy to see which objects have the highest deep size (rather than shallow size which the normal tools show) – Peter Lawrey Nov 09 '13 at 07:47
Ok I will try to go and install it. But that require a license key rite. Fro now I will try it out the trial license first. IS there any better alternative to this have you heard of java melody ? – user2711681 Nov 09 '13 at 09:48
I managed to install yourkit. Yes its offer more information and quite in details. I went into biggest obbject(dominators) is all showing me java.lang.ref.finalize. I keep clicking the + it goes deeper. So how to find the root cause of it ? Any idea should I be looking else where I also went into object explorer showing me same thing too – user2711681 Nov 10 '13 at 07:31
The object in question is the `referent` You need to look at what these objects are. – Peter Lawrey Nov 10 '13 at 15:03
I went into reachability scopes and in the object explorer is where I saw the referent option when I put the + button. There a lot of it but mostly are either referent as java.io.FileOutputStream or java.net.SocksSocketImpl. If you notice in my code snippet in my question I have a finally which closes the socket. So what else is causing it to refer to socket? I do have a function which write to a text file where I declare in function this FileWriter fw = null; and finally I do this if ( fw != null ){fw.close();} – user2711681 Nov 11 '13 at 13:06
@user2711681 Unfortunately you only need one object which is slow to close to block the whole queue. Can you try work out which object the finalizer thread is working on. Not sure its that easy with a profiler, but you should be able to with a debugger by stopping the finalizer thread periodically. – Peter Lawrey Nov 11 '13 at 13:28
I dont get you when you said you debugger to stop the finalizer thread periodically do you mean I need to do it via my codes intervention now? – user2711681 Nov 11 '13 at 13:33
@user2711681 debug the program and when you think the finalizer queue is high, pause the process with the pause button in your IDE and look at what the finalizer thread is doing and trying to clean up. Or you could get a stack dump of this thread periodically to see what it is doing. – Peter Lawrey Nov 11 '13 at 13:36
my application does not have GUI and it running remotely on a linux machine. So I am not using the IDE to run. I guess I need to get a stack dump is this the right command to get the stack jstack -l ? What should I be looking in it ? – user2711681 Nov 11 '13 at 13:42
You are looking for the stack trace of the Finalizer Thread. i.e. what is it doing and why isn't it clearing the queue faster? BTW You can do a remote debug from you IDE to do the same. – Peter Lawrey Nov 11 '13 at 14:07
The problem is that the one object which you can't find in the queue is the one being closed and if you have a rare slow finalizer that is the one you need to see. – Peter Lawrey Nov 11 '13 at 14:09
I am using actually jcreator. I have a bad experience setting up before this even the visual vm to remotely connect to my machine. So that is where I opt to take frequent jmap snapshots. So do you think jstack will be helpful ? What is best you suggest? – user2711681 Nov 11 '13 at 14:27
I did a simple jstack command I can see these "com.google.common.base.internal.Finalizer" daemon prio=10 tid=0x00007fac1454e800 nid=0x739a in Object.wait() [0x00007fac1963e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) and "Finalizer" daemon prio=10 tid=0x00007fac1408c000 nid=0x7386 in Object.wait() [0x00007fac1a7d1000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) – user2711681 Nov 11 '13 at 15:33
which queue are you referring is the jstack indicating some problem or issue here? – user2711681 Nov 11 '13 at 17:34
@user2711681 You keep saying there is lots of Finalizer objects retained, these are in one Queue for the finalizer thread. – Peter Lawrey Nov 11 '13 at 20:57
/jmap -finalizerinfo 29570 it gives me JVM version is22.1-b02 Number of objects pending for finalization: 0. My jstack do have these"com.google.common.base.internal.Finalizer" daemon prio=10 tid=0x00007fd5b858d000 nid=0x6fe in Object.wait() [0x00007fd5ae781000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) and "Finalizer" daemon prio=10 tid=0x00007fd5b809c800 nid=0x6e9 in Object.wait() [0x00007fd5af6f5000] java.lang.Thread.State: WAITING on object monitor. Jmap results show java.lang.ref.Finalizeroccupy 14,321,696(32.46%)bytes? – user2711681 Nov 12 '13 at 08:19
So how can I translate the 3 above different statistics where /jmap -finalizerinfo say 0 objects pending for finalization but then jmap result of heap dump shows java.lang.ref.Finalizeroccupy 14,321,696(32.46%)bytes? Together also the jstack shows instance of finalizer there ? – user2711681 Nov 12 '13 at 08:21
Can you do a `jmap -histo:live {pid} | head -22` to check these finalizers are actually consuming live memory as it appears they are not? – Peter Lawrey Nov 12 '13 at 13:59
I have updated my question with the results of jmap -histo:live {pid} | head -22 .I also got one gc happened so I google and it say this command will trigger gc. So I tried one more time yes I got my second gc now and good I notice immediately after my gc the old is now 16.22 and perm is drop from 81.90 previously to just 43.82. Actually how will the histo live will be helpful ? – user2711681 Nov 12 '13 at 16:33
since I ran this command jmap -histo:live {pid} | head -22 twice I have been taking few heap dump and now the only one taking the biggest memory is the bonecp pooling. The finalizer is not reporting. What is opinion with regards to the jmap -histo:live {pid} | head -22 results which I have posted earlier. – user2711681 Nov 14 '13 at 08:53
@user2711681 Looking down you list a class which stands out is the ResultSet, which suggest you are not closing them properly and the Socket seems pretty high, possibly another memory leak. – Peter Lawrey Nov 14 '13 at 15:46
if you look at my code snippet towards the end finally block I have this try{ if ( writeBuffer != null ){writeBuffer.close();} else{ System.out.println("w is null in finally close");}}. Why I close the write buffer because I read few places that by just closing that will close the whole socket. Do this is contributing to the leak ? I dont get you from list which one is pointing to the ResultSet? For every resultset I have ensured this try{if ( rs2!= null ){rs2.close(); }else{System.out.println("No rs2 exist"); }if ( stmt2!= null ){stmt2.close();}} – user2711681 Nov 14 '13 at 17:08
@user2711681 And yet they are being retained. There is a resource you are not closing. – Peter Lawrey Nov 14 '13 at 17:14
but what you think about the Socket is it the right way I am closing it should I close using other mechanism not just the write buffer ? How about the resultset the way I am closing is it acceptable ? Yes looking at the histogram by right both of these values should be very minimal right? – user2711681 Nov 14 '13 at 17:25
@user2711681 They should be minimal, but they are not, they are 10s of thousands which means that somewhere in your code you are not closing those resources. You might be doing right in one place, but it has to be right every where. – Peter Lawrey Nov 14 '13 at 17:48
@I will check through every bit of the codes. But are my method of closing both socket n resultset conform to the standards or out ouf bound ? What I am worried is the way I close might not be right any input on that ? – user2711681 Nov 14 '13 at 17:53
over the 2 days I have been monitoring with few samples of the jmap live histo what I notice is that this value com.mysql.jdbc.JDBC4ResultSet is fluctuate at times is up then down. But this one java.net.SocksSocketImpl it keep going higher and higher. There is only one place where I close the socket. Do you think anything is leading to his? – user2711681 Nov 16 '13 at 10:04
Even though the socket is closed, you might be retaining them in a list/collection. – Peter Lawrey Nov 16 '13 at 11:59
I dont retain any list of socket. Could that be at the java implementation itself? How to clear both this socket n resultset fully. Just to update I have gone through in detail my codes where I do a find each ResultSet and make sure that there is a closing for it and its statement too. Just to add I did I another test on a smaller machine. I strip off everything just remain the socket codes. I have updated before and after running results of the pure socket in my question. You will notice java.net.SocksSocketImpl and java.net.Socket have gone up from the initial value. – user2711681 Nov 16 '13 at 15:24
any comment on the retain values which I have posted in my question ? – user2711681 Nov 19 '13 at 08:36
what do you say should I clear the resultset and socket in different manner? – user2711681 Nov 21 '13 at 16:02

GC and YGC occurrence and next action of course

1 Answers1