3

We have two machines with identical configuration and use (we have two balanced Siebel application servers in them).

Normally, we have a very similar RAM usage in them (around 7 Gb). Recently, we've have a sudden increase of RAM in only one of them and now we have close to 14 Gb utilization of RAM in that machine.

So, for very similar boxes, we have one of them using 7Gb of RAM while the other one is consuming 14 Gb.

Now, using ps aux command to determine which process it's using all this additional memory, we see memory consumption is very similar in both machines. Somehow, we don't see any process that's using those 7 Gb of additional RAM.

Let's see: Machine 1: total used free shared buffers cached Mem: 15943 15739 204 0 221 1267 -/+ buffers/cache: 14249 1693 Swap: 8191 0 8191

So, we have 14249 Mb usage of RAM.

Machine 2: total used free shared buffers cached Mem: 15943 15636 306 0 962 6409 -/+ buffers/cache: 8264 7678 Swap: 8191 0 8191

So, we have 8264 Mb usage of RAM.

I guess, the sum of Resident Set Size memory of ps should be equal or bigger to this value. According to this answer is how much memory is allocated to the process and is in RAM (including memory from shared libraries). We don't have any memory in SWAP.

However:

Machine 1:

ps aux | awk 'BEGIN {sum=0} {sum +=$6} END {print sum/1024}' 8357.08

8357.08 < 14249 -> NOK!

Machine 2:

ps aux | awk 'BEGIN {sum=0} {sum +=$6} END {print sum/1024}' 8468.63

8468.63 > 8264 -> OK

What do I get wrong? How can I find where this "missing" memory is?

Thank you in advance

user3779018
  • 65
  • 2
  • 8
  • The ps figure won't take account of shared memory between process and so comparing summed data from ps with free is difficult. – Raman Sailopal Aug 02 '17 at 11:20
  • 1
    That's right, Raman but, as a consquence, I would exepect ps ouptut to be bigger that the one from free not the other way around. – user3779018 Aug 02 '17 at 11:26
  • Free includes shared memory between processes and ps doesn't. Free therefore is normally bigger. – Raman Sailopal Aug 02 '17 at 11:31
  • Thank you for your insight Raman, however I understand: "Resident Set Size is the amount of physical memory currently allocated and used by a process (without swapped out pages). It includes the code, data and shared libraries (which are counted in every process which uses them)". See http://trustmeiamadeveloper.com/2016/03/18/where-is-my-memory-java from Mikhail Krestjaninoff. – user3779018 Aug 02 '17 at 14:01
  • @user3779018 Just met the same problem, do you have any progress? – a.l. Apr 17 '18 at 07:28
  • I have same issue, did you find a solution/way to troubleshoot further? – Czar Feb 20 '21 at 19:31

1 Answers1

0

If them two are virtual machines, maybe the "missing" memory is occupied by Balloon driver, especially they are hosted by VMware ESXi.

Recently I encounter the similar scenario. Sum of all process RSS is 14GB, command free shows 26GB used, so there are 12GB memory missing.

After search on internet, I follow this article and execute command vmware-toolbox-cmd stat balloon on my VM, console shows 12xxxMB (used by balloon), BINGO!

Wayne
  • 26
  • 5