Malloc allocates memory more than RAM

Question

I just executed a program that mallocs 13 MB in a 12 MB machine (QEMU Emulated!) . Not just that, i even browsed through the memory and filled junk in it...

void 
large_mem(void) 
{
  #define LONGMEM  13631488
  long long *ptr = (long long *)malloc(LONGMEM);
  long long i;
  if(!ptr) {
     printf("%s(): array allocation of size %lld failed.\n",__func__,LONGMEM);
     ASSERT(0);
  }
  for(i = 0 ; i < LONGMEM ; i++ ) { 
    *(ptr+i)=i;
  }
  free(ptr);
}

How is it possible ? I was expecting a segmentation fault.

Woah! just realised that the code inside the forloop is wrong... still still still..! Why no seg fault ? — raj, Sep 21 '11 at 17:41

score 12 · Accepted Answer · answered Sep 21 '11 at 17:40

12

It's called virtual memory which is allocated for your program. It's not real memory which you call RAM.

There is a max limit for virtual memory as well, but it's higher than RAM. It's implemented (and defined) by your operating system.

answered Sep 21 '11 at 17:40

Nawaz

353,942
115
666
851

so there's no max limit or something ? – raj Sep 21 '11 at 17:42
will the new pages gets swapped in as i keep requesting the OS ? – raj Sep 21 '11 at 17:43
3

@raj: The theoretical limit is determined by the size of a pointer. Practically, most OS will give you less memory than whatever address space the pointer size yields. And, yes, if you allocate more memory than physically present in the machine, you will get swapping. – sbi Sep 21 '11 at 17:44
woah! its like i'm chatting with you guys.. such fast replies.. C community is sooo fast. :) thanks a lot all.. – raj Sep 21 '11 at 17:47
its stupid that i have to wait 10 minutes to aceept ur answer in SO.. :P – raj Sep 21 '11 at 17:51
This answer is rather incomplete without mentioning overcommit. On a system configured to manage commit charge, `malloc` would fail here. – R.. GitHub STOP HELPING ICE Sep 21 '11 at 17:55
@raj: [There's a reason why you have to wait before accepting an answer](http://meta.stackexchange.com/questions/47448/allow-me-to-accept-an-answer-immediately/75200#75200). – In silico Sep 21 '11 at 17:55
@ln silico. poor guy.. asked this question and got -14 votes :P – raj Sep 21 '11 at 18:00

score 9 · Answer 2 · edited Sep 21 '11 at 17:51

This is called as Lazy Allocation.

Most OS like Linux have an Lazy Allocation memory model wherein the returned memory address is a virtual address and the actual allocation only happens at access-time. The OS assumes that it will be able to provide this allocation at access-Time.

The memory allocated by malloc is not backed by real memory until the program actually touches it.

While, since calloc initializes the memory to 0 you can be assured that the OS has already backed the allocation with actual RAM (or swap).

Try using callocand most probably it will return you out of memory unless your swap file/partition is big enough to satisfy the request.

In silico · Answer 3 · 2011-09-21T17:47:01.313

Sounds like your operating system is swapping pages:

Paging is an important part of virtual memory implementation in most contemporary general-purpose operating systems, allowing them to use disk storage for data that does not fit into physical random-access memory (RAM).

In other words, the operating system is using some of your hard disk space to satisfy your 13 MB allocation request (at great expense of speed, since the hard disk is much, much slower than RAM).

score 4 · Answer 4 · answered Sep 21 '11 at 17:54

Unless the virtualized OS has swap available, what you're encountering is called overcommit, and it basically exists because the easy way to manage resources in a system with virtual memory and demand/copy-on-write pages is not to manage them. Overcommit is a lot like a bank loaning out more money than it actually has -- it seems to work for a while, then things come crashing down. The first thing you should do when setting up a Linux system is fix this with the command:

echo "2" > /proc/sys/vm/overcommit_memory

That just affects the currently running kernel; you can make it permanent by adding a line to /etc/sysctl.conf:

vm.overcommit_memory=2

BTW, I have provided a concrete example that invokes the out of memory killer at: https://stackoverflow.com/questions/7880784/what-is-rss-and-vsz-in-linux-memory-management/57453334#57453334 — Ciro Santilli OurBigBook.com, Aug 28 '19 at 07:55

Malloc allocates memory more than RAM

4 Answers4

Linked