1

I have a microservice like java framework. Many java processes run on a single box (ubuntu 14.04.4 LTS). The java processes use alot system memory, so the swap space is used heavily. The jstat gc reports do not explain the systems memory usage. All java processes run with the parameters

-XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90

to force the JVM to give memory back to the system. Without the parameters the problem persists. Some java components use nashorn engine to script some functionality.

Can someone explain the behavior here?

Are there any jvm patameters that restrict the huge system memory usage?

How to command the OS to be more restrictive with memory allocation for the jvm?

Some data:

Component A (with nashorn)

top:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
2400 xxxxxx    20   0 13.933g 807496   7332 S   0.0  2.5   4180:15 java

jstat -gc 2400:

S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
512.0  512.0   0.0   400.0  19456.0  12751.6   62464.0    59862.3   89688.0 84866.6 10624.0 9440.4 2165265 15977.896 16816 1813.836 17791.732
  • capacity: ca. 180 MB
  • usage: ca. 165 MB
  • system resources: ca. 800 MB

Why does the component uses more the 4 times the GC areas memory?

Component B (without nashorn)

top:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
19476  xxxx     20   0 13.465g 120436   7836 S   7.0  0.4  22:40.76 java

jstat -gc 19476:

S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
512.0  512.0   0.0    0.0   41472.0  25408.7   343040.0    7164.5   17664.0 17183.1 2048.0 1919.4   3650   10.806  939    16.788   27.594
  • capacity: ca. 403 MB
  • usage: ca. 52 MB
  • system resources: 120 MB

Here the GC area capacities are bigger than the actual systems memory usage. Still the systems memory usage is twice the GC areas. IMO this component behaves normal because the libraries etc. are mapped partially into memory as well.

Component C (without nashorn)

top:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
2272 xxxxxx    20   0 13.382g 922944  11108 S   0.7  2.8  40033:41 java

jstat -gc 2272:

S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
1024.0 1024.0 868.0   0.0   36352.0  23866.1   76800.0    56580.2   68864.0 64571.1 8448.0 7460.6 31974159 199295.501 844692 134644.040 333939.541
  • capacity: ca. 190 MB
  • usage: ca. 152 MB
  • system resources: 920 MB

Why does the component uses more the 6 times the GC areas memory?

J. Mueller
  • 13
  • 3
  • 2
    Not helpful, I know, but still. You know the first rule of distributed computing? *Don't distribute computing*. I understand that having a lot of small services can solve a lot of problems, but isn't the idea that it should scale on your environment. I kinda don't get the idea behind using so many different JVMs (on the same node) that you run into such problems in the end ... wouldnt the correct answer be either to use more than one node, or use less JVMs? – GhostCat Jul 06 '16 at 12:20
  • I would suggest looking at the answer given in: http://stackoverflow.com/questions/14763079/what-are-the-xms-and-xmx-parameters-when-starting-jvms – Vishal Jumani Jul 06 '16 at 12:25
  • It has something todo with the jvm memory management on linux/ubuntu.Running on windows I do not observe those problems. The Xmx/Xms flags restrict the heap sizes but have less influence on the system memory usage. – J. Mueller Jul 06 '16 at 14:16

1 Answers1

3

Can someone explain the behavior here?

There is no single explanation for memory usage, there are many contributing factors.

  • Use NMT to get an overview of how much memory is allocated by various internal parts of the JVM
  • use pmap -x <pid> to identify memory-mapped files.
  • take a heap dump to look for direct memory buffers (they just show up as some of the [anon] mappings in pmap) or use Yourkit which has a memory inspection to identify the amount allocated by direct buffers. At runtime you can use BufferPoolMXBean to track direct buffer use.

Beyond that, you have to take into account that each JVM comes with some baseline memory consumption and needed breathing room for the garbage collector. Running multiple services in a shared JVM can amortize those baseline costs.

Due to complexities of the virtual memory system you also need to be aware of the difference between used, committed, reserved and resident memory.

Are there any jvm patameters that restrict the huge system memory usage?

That would depend on the cause.

For the managed heap it's possible to make it yield unused memory back to the OS more swiftly, but comes with a performance penalty.

the8472
  • 40,999
  • 5
  • 70
  • 122