1

In my C program, based on the user's input, memory will be allocated for a given simulation. The initial problem I faced is that user can ask for a huge number to allocate but malloc() never fails until it runs out of memory then the program crashes.

I investigated the logic behind this and it now makes sense to me, see [1][2]. A possible workaround given here "SIGKILL while allocating memory in C++" suggests to set overcommit_memory in in /proc/sys/vm/overcommit_memory from 0 to 2.

This solved the problem from one side. But since I am using -fsanitize=address I get error from sanitizer.

Is there any better solution to this?

Community
  • 1
  • 1
Pourya
  • 21
  • 4
  • 2
    If the user gives the value `X` as input, you *know* how many bytes that will result in your program allocating. You could easily limit the value of `X` to something that will not exhaust the system. So if the user inputs a to large value, just tell the user it's to large and ask for the value again. – Some programmer dude Sep 27 '16 at 09:32
  • 1
    Also, you *do* know that [`malloc`](http://en.cppreference.com/w/c/memory/malloc) returns `NULL` if it fails to allocate memory? You could also easily check for that and not attempt to dereference the null pointer. – Some programmer dude Sep 27 '16 at 09:35
  • Please provide an Minimal, Complete, and Verifiable example (http://stackoverflow.com/help/mcve) – stenliis Sep 27 '16 at 09:53
  • 5
    @JoachimPileborg I think he means that `malloc()` overcommits, so it doesn't return `NULL` although there's no physical memory to actually back the allocation, so the kernel kills the process when it tries. – unwind Sep 27 '16 at 10:22
  • 1
    @JoachimPileborg I dont know the specs of the user's computer and I don't want to limit the program based on my laptop. This won't solve my problem. – Pourya Sep 27 '16 at 14:49
  • And as unwind mentioned, malloc() doesn't return NULL. If i continue allocating, it gives me a 'physical address' and it does this until the kernel stops the process. – Pourya Sep 27 '16 at 15:00
  • Can you try `calloc()` instead of `malloc()`? Maybe `calloc()` does return `NULL` when asked to allocate **and initialize** too much space. – pmg Sep 27 '16 at 15:04
  • 1
    @pmg: calloc generally doesn't initialise most of the allocation - it typically just sets all unmodified pages to point at a read-only zero page and then wires pages on demand whenever there is a page fault due to a write. This makes it very efficient for sparse allocations, but it doesn't help with the OP's problem. – Paul R Sep 27 '16 at 15:08
  • @pmg I also tried with calloc() which resulted in the same problem. – Pourya Sep 27 '16 at 15:16
  • Then how about finding out how much memory the current system have? Even if there's no system call to do it, you can always just parse `/proc/meminfo`. – Some programmer dude Sep 27 '16 at 17:48
  • I don't know your allocation pattern, but what about asking the user how much RAM to use on the simulation, pre-allocate that amount at startup and use from that pool instead of calling `malloc` every time you need memory? That will give you the opportunity to tell the user that the program has run out and exit gracefully. – Morten Jensen Sep 28 '16 at 15:39

1 Answers1

0

I guess the clang AddressSanitizer is failing because there is a legit leak. So my answer ignores that:

Alternatives:

  1. Disable the overcommit behaviour, as you have already figured out: that is going to affect other processes and requires root.
  2. run you app in a docker image with the oom killer disabled: that doesn't affect other processes but requires root to install docker (this is my favourite solution though).
  3. write after malloc: may take long to alloc a huge chunk of memory and your process can still get killed because of other running process but doesn't require root.
  4. use ulimit -v to limit the amount of memory depending on the machine: that also doesn't require root but your process might be killed anyway.

Code for the third alternative (for linux):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <setjmp.h>

jmp_buf resume_malloc;

void handle_malloc_error(int sig)
{
   longjmp(resume_malloc, sig);
}

void *memalloc(size_t sz) {
   void *p = 0;
   int sig = setjmp(resume_malloc);
   if ( sig == 0 ) {
      p = malloc(sz);
      signal(SIGSEGV, &handle_malloc_error);
      memset(p, 0, sz);
   } else {
      p = 0;
   }
   signal(SIGSEGV, SIG_DFL);
   return p;
}

int main(int argc, char *argv[])
{
   size_t sz = 160L * 1024 * 1024 * 1024L;
   void *p;
   for (int i=0; i < 100; i++) {
      printf("size: %lu\n", sz);
      p = memalloc(sz);
      if ( p == 0 ) {
         printf("out of memory\n");
         break;
      }
      sz *= 2;
   }
}
olivecoder
  • 2,858
  • 23
  • 22
  • Thank you for the great explanation! I really liked the solution using long jump. I have tried that, but I still face the same problem. So the process gets killed. About the sanitizer: There is no leak. I should have explained this better. The sanitizer cannot 'initialize' and just after I run the program I get " ReserveShadowMemoryRange failed while trying to map 0xdfff0001000 bytes. perhaps you are using ulimit -v". The root options are not applicable in my case because the code will be used on other computers. I would appreciate any other suggestions. – Pourya Sep 29 '16 at 07:51
  • One more remark that the address sanitize works fine when overcommit_memory is 0 – Pourya Sep 29 '16 at 07:57
  • So it seems like other process is trying to write to a memory that isn't actually available and your process is being selected by the OOM killer to be killed. If so you need the first or second solution. – olivecoder Sep 29 '16 at 10:49
  • Unfortunately I don't actually know how the sanitizer is affected by the overcommit settings. I'd need to test that and you could get better answers if you provide a small code snippet to reproduce the problem. – olivecoder Sep 29 '16 at 10:51