I am running two tomcat servers in a single machine(Ram size 16GB), with the memory setting given below.
JAVA_MEM_OPTIONS="-Xmx4096M -Xms4096M -XX:NewSize=512M -XX:MaxNewSize=512M  -XX:PermSize=256M -XX:MaxPermSize=256M -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails  -XX:+UseConcMarkSweepGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:SoftRefLRUPolicyMSPerMB=5 -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC"
Top result shows 46.5%(7.44 GB) & 39.2%(6.24 GB) of memory are used by java process.
Tasks: 120 total,   1 running, 119 sleeping,   0 stopped,   0 zombie
Cpu(s):  7.8%us,  0.4%sy,  0.0%ni, 90.9%id,  0.2%wa,  0.0%hi,  0.7%si,  0.0%st
Mem:  16424048k total, 16326072k used,    97976k free,    28868k buffers
Swap:  1959920k total,  1957932k used,     1988k free,  1082276k cached
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                         
26275 user1      25   0 8007m 7.3g 7832 S 13.6 46.5 785:31.89 java                                                                                                                                            
28050 user2      25   0 7731m 6.1g  13m S  9.0 39.2 817:10.47 java  
GC log shows each java process is using only 4GB of memory. Once the system memory usage goes up its never coming down.
user1 - java process
89224.492: [GC 89224.492: [ParNew: 443819K->17796K(471872K), 0.0367070 secs] 2032217K->1608465K(4141888K), 0.0368920 secs] [Times: user=0.13 sys=0.00, real=0.03 secs] 
89228.247: [GC 89228.247: [ParNew: 437252K->22219K(471872K), 0.0607080 secs] 2027921K->1615327K(4141888K), 0.0609240 secs] [Times: user=0.15 sys=0.00, real=0.06 secs] 
user2 - java process
89202.170: [GC 89202.170: [ParNew: 444989K->22909K(471872K), 0.0510290 secs] 2361057K->1945258K(4141888K), 0.0512370 secs] [Times: user=0.19 sys=0.00, real=0.05 secs] 
89207.894: [GC 89207.894: [ParNew: 442365K->15912K(471872K), 0.0422190 secs] 2364714K->1945162K(4141888K), 0.0424260 secs] [Times: user=0.15 sys=0.00, real=0.04 secs]
How java process can use this much memory when heap memory is set to 4GB. How to debug the cause for the problem.
PS : At times i am executing shell script from java code. Will it leads to this sort of problem ?
 
     
    