Is it possible to make the OOM killer intervent earlier?

Question

I try to tweak my development system to maximal reliability. I disabled swap, because for GUI usage it mostly renders the machine unresponsive in such a way not useable anymore. Nevertheless, if agressive appications eat up the memory, some mechanisms seem to kick in that making the most out of it on cost of speed. There is no harddrive swap operation, but the system is getting unresponsive likewise. So I want to let the OOM killer kick in before the system make any special efforts on memory gain. Is it possible to configure the OOM killer to act if there is less than 100 MB free physical memory for example?

Answer 1

I also struggled with that issue. I just want my system to stay responsive, no matter what, and I prefer losing processes to waiting a few minutes. There seems to be no way to achieve this using the kernel oom killer.

However, in the user space, we can do whatever we want. So I wrote the Early OOM Daemon ( https://github.com/rfjakob/earlyoom ) that will kill the largest process (by RSS) once the available RAM goes below 10%.

Without earlyoom, it has been easy to lock up my machine (8GB RAM) by starting http://www.unrealengine.com/html5/ a few times. Now, the guilty browser tabs get killed before things get out of hand.

Answer 2

For me setting vm.admin_reserve_kbytes=262144 does exactly this thing. OOM killer intervents before system goes completely unresponsive.

Answer 3

The default policy of the kernel is to allow applications to keep allocating virtual memory as long as there is free physical memory. The physical memory isn't actually used until the applications touch the virtual memory they allocated, so an application can allocate much more memory than the system has, then start touching it later, causing the kernel to run out of memory, and trigger the out of memory (OOM) killer. Before the hogging process is killed though, it has caused the disk cache to be emptied, which makes the system slow to respond for a while until the cache refills.

You can change the default policy to disallow memory overcommit by writing a value of 2 to /proc/sys/vm/overcommit_memory. The default value of /proc/sys/vm/overcommit_ratio is 50, so the kernel will not allow applications to allocate more than 50% of ram+swap. If you have no swap, then the kernel will not allow applications to allocate more than 50% of your ram, leaving the other 50% free for the cache. That may be a bit excessive, so you may want to increase this value to say, 85% or so, so applications can allocate up to 85% of your ram, leaving 15% for the cache.

Answer 4

The other answers have good automatic solutions, but I find it can be helpful to also enable the SysRq key for when things get out of hand. With the SysRq key, you'd be manually messaging the kernel, and you can do things like a safe reboot (with SysRQ + REISUB) even if userspace has completely frozen.

To allow the kernel to listen to requests, set kernel.sysrq = 1, or enable just the functions you're likely to use with a bitmask (documented here). For example kernel.sysrq = 244 will enable all the combos needed for the safe reboot above as well as manual invocation of the OOM killer with SysRq + F.

Answer 5

Reliability isn't reached by low memory conditions and an OOM killer.

It is wrong to organize a party in a closet and place "cleaning out my closet" on your small playlist.

Is it possible to make the OOM killer intervent earlier?

Doing this will have unintended side results, because you have no control over what is killed.

I try to tweak my development system to maximal reliability.

Maximal reliability involves testing your system and improving your system based on these tests.

Just tweaking random things won't get you anywhere...

I disabled swap, because for GUI usage it mostly renders the machine unresponsive in such a way not useable anymore. Nevertheless, if agressive appications eat up the memory, some mechanisms seem to kick in that making the most out of it on cost of speed.

Due to low memory conditions, disabling the swap won't improve the behavior, it does the opposite.

To increase reliability in this situation, add more memory such that your system is more responsive and there are no random processes being killed without the user's intention. You shouldn't resort on low memory conditions and a mechanism like this, especially not in a development environment...

There is no harddrive swap operation, but the system is getting unresponsive likewise.

Low memory conditions indeed result in unresponiveness, whether you have a swap or not.

So I want to let the OOM killer kick in before the system make any special efforts on memory gain.

Special efforts that will do more harm than good, as I explained above. Instead, you could kill processes you don't need yourself, but I guess you can't do that so the OOM will kill processes that you need.

Is it possible to configure the OOM killer to act if there is less than 100 MB free physical memory for example?

Might be, but you get a higher return on investment if you just buy some extra memory which doesn't really cost much these days. Consider that you're going to hit yourself in the foot on the long run if you continue to work on low memory conditions. OOM is like a bailiff, it doesn't assist you, it assists the OS...