Is gradually growing swap usage an indicator for a memory leak?

Question

I’m trying to understand whether I have a memory leak on my server (Windows Server 2012R2). The server has 48 GB RAM and was designed to run only a DBMS (a SQL Server 2014) but there are also other tools like Symantec Endpoint Protection, FireEye and some monitoring tools/services installed. It is currently configured to reserve 42 GB to DBMS leaving 6 GB to OS and other services. There were out-of-memory errors when only 4 GB were reserved for OS and other services, which disappeared after I increased this to 6 GB. High RAM usage itself is normal for a SQL Server machine, as SQL Server buffers the data in RAM.

What worries me is that swap usage (performance counter Paging File->Usage) is gradually increasing in the last 2 months currently reaching 15 % of 24 GB (size of the swap file). I’m not sure whether this is fine because Windows puts stuff in swap even under normal conditions or is it an indicator that 6 GB also won’t be enough after some time. On the chart below green line is RAM usage (spikes down when I restarted the SQL Server and rapidly gets up again) and yellow is the swap usage since mid-May.

Swap usage image

What I have done:

1. Dumped the performance counters “Process->Private bytes”. Nothing really suspicious here. TOP10 in MBs are (total 44711 MB):

   • sqlserver 41322
   • xagt#5 504
   • ccsvchst 316
   • splunkd 285
   • ccsvchst#1 260
   • monitoringhost 233
   • xagt#1 151
   • monitoringhost#1 129
   • healthservice 116
   • svchost#4 110

2. Tried to restart individual services (SQL Server, xagt/Fireeye, splunkd). This had no effect on the swap usage.

3. Tried to get more information using tools RAMMap from Sysinternals and VMMap as I suspected that some process leaks memory through memory mapped files, which don’t count to private bytes.

   RAMMap reports following (unfortunately the files under File Summary/File Details are only adding up to less than 300 MB used though 2659 MB reported as active for mapped files):

   Screenshot from rammap

   

I have another server with the same tools installed, but 96 GB RAM and 16 GB swap. RAM usage is below 50 %, but swap usage is also (almost) continuously increasing currently at 28 %.

Swap usage on another server

Is this kind of behavior normal? If not, how I can find the problematic process?

Answer 1

Backing swap storage is a parameter that is consultable. When it's sufficient then over-commitment of memory is not required. My previous answer was not the real explanation.

The conclusion is then that the poster has a real memory leak that doesn't show up in the tools. This usually means that the leak is in Windows or some third-party driver.

Such leaks are very hard to find, and when found they are often in proprietary code that cannot be modified.

As the leak in this case is very small with no performance hit, the problem is mostly just an annoyance. It can be fixed by a reboot every so often. In the poster's case this can even be something as infrequent as once a month.

---

Previous answer when less information was available:

You are apparently under the impression that the swap space is only used when Windows (or any OS) runs out of RAM. This is far from the case.

The OS must be ready to swap out processes when RAM is exhausted. For that reason, for any process that asks for RAM, the OS allocated backing swap space that is equal to the amount requested, and allocates it even before it grants the RAM request. The allocated swap space is not necessarily allocated as specific disk sectors, but the OS will make sure that enough free space exists in case of need.

The increase you observe means that your DBMS is slowly increasing its memory size and asking for more RAM. The swap increases accordingly.

To be totally safe, the swap space should equal the RAM size. The 6 GB you allocated are far from sufficient. Increased swap usage should not be alarming.

Answer 2

I think this is because of:

when something has been paged out and later read back in, the kernel will not remove it from the page file until either the file is otherwise full or the page in RAM is altered. This way if it needs to page that chunk out again it can do so without having to actually write the pages to disk: the content is already there

from this answer.

Also from a comment there, Microsoft recommends your swap be large enough that it is only 50-75% used at any given time...which is not a problem here.

Microsoft also recommends (from same link):

Memory, Pages Output/Sec: This counter shows how many virtual memory pages were written to the pagefile to free RAM page frames for other purposes each second.

This is the best counter to monitor if you suspect that paging is your performance bottleneck. Even if Committed Bytes is greater than the installed RAM, if Pages Output/sec is low or zero most of the time, there is no significant performance problem from insufficient RAM.

Short version: modern OSes are like hoarders. They will only let go of something when forced to (hits some limit and cleans out old stuff, or something changed so it lets go of the out of date copy) basically because it can and it might be useful some day...

Answer 3

This could be a good thing, a sign of a server OS working properly, using swap space to free up RAM for disk caching. When the OS first starts up, a bunch of processes are running, but probably never (or rarely) do anything useful because you aren't using those parts of Windows. Their memory can get paged out, but that only happens gradually when the OS happens to try to reclaim a page.

In low-memory-pressure situations, it's normal for an OS to occasionally pick a random page and see if it's in use, e.g. by marking it invalid (but leaving it in memory so it can quickly wire it back in to the page table for the process that owns it). If it doesn't get referenced any time soon, the OS might page it out to disk. (And reclaim it right away, or leave it in RAM until there's a need for more pages for something.)

In Linux, the aggressiveness of this process is set by vm.swappiness. I don't know if Windows has a similar tuning setting.

---

15% of your 24 GB swap file is only 3.6 GB, well under the 6GB size you reserved for everything but the DBMS.

Hopefully, your OS is just paging out dirty pages that never get modified, when some random memory pressure happens to identify such a page. Such pages are probably common in programs that start up but don't really have anything to do when nothing connects to their services.

The best case is when such a page never even gets read, so it can stay out or RAM. Then your swap space is doing its job, leaving more RAM available for disk caching and other useful stuff.

But @user3067860's answer provides another very useful clue: Windows will leave a page in swap space even if it's read back in, so it can be evicted from RAM again for free, without I/O. It only gets dropped from swap if the page is modified (or freed I guess, e.g. by restarting whatever process owned it).

In most processes that just sit there waiting for someone to talk to them, they probably allocated some memory and wrote to it during startup, but now are just sitting in an event loop waiting for an event that never comes. Or waking up every few seconds or once a day for something, maybe touching some stack space and a page or two.

But the rest of their pages that they dirtied during startup are not going to be written again ever, and most of them aren't even going to be read if whatever Windows service isn't needed.

So long-term, we'd expect that most of those pages (stack and private .data/.bss pages, and dynamic allocations, "heap") will eventually get paged out. Any that are read but not written will still end up in swap space eventually, but also stay hot in RAM.

And of course there are some things other than your DBMS that are doing stuff, like presumably Windows update and logging, so there's some scope for growing the total memory footprint beyond what it was right after boot.

---

This answer is a guess based on how OSes are designed in general, zero experience with actual Windows server tuning. There might be an actual problem, but it's also quite possible your swap usage will plateau if you leave it alone. I'd keep an eye on it, but I wouldn't worry unless your swap usage gets over 6 GB. If it gets to 12 GiB, then maybe schedule a reboot.

Since restarting your DBMS didn't affect swap usage, it seems those pages weren't from it. That makes sense; it's actually doing real work and probably writing most of its memory pages frequently. And your partitioning of physical RAM maybe makes Windows not try to reclaim its pages.