Why can I allocate arrays that are larger than my RAM?

Question

I am facing a weird issue with malloc calls. I am working on a program that uses huge arrays (sizes in GBs) and while trying to allocate memory for the array using malloc I find that, malloc is successful even when I allocate a size that is bigger than my RAM (which is 64GB).

See code below:

#include<stdio.h>
#include<stdlib.h>
#include<malloc.h>


#define Sixteen_G  16000000000

int main() {
    int *c = (int *)malloc(sizeof(int)*Sixteen_G);
    int *d = (int *)malloc(sizeof(int)*Sixteen_G);
    int *e = (int *)malloc(sizeof(int)*Sixteen_G);
    int *f = (int *)malloc(sizeof(int)*Sixteen_G);
    int *g = (int *)malloc(sizeof(int)*Sixteen_G);
    if(c == NULL)
            printf("c Allocation failed\n");
    if(d == NULL)
            printf("d Allocation failed\n");
    if(e == NULL)
            printf("e Allocation failed\n");
    if(f == NULL)
            printf("e Allocation failed\n");
    if(g == NULL)
            printf("e Allocation failed\n");
    else
            printf("All arrays allocated\n");
    return 0;
}

The output for the above code is:

All arrays allocated

Also a single call to malloc of size >= 17GB is failing, but multiple calls to 16GB are passing.

Can someone explain why malloc is able to allocate that much memory, when my system RAM size is just 64GB, and also how exactly malloc works on single/multiple calls

The `else` only belong to the `if(g == NULL)` check, not the others. — Some programmer dude, Nov 12 '17 at 18:59
Someone might talk to you about how the particular `malloc` implementation on your system does those things, if you tell us what system that is. The C language specifications do not themselves speak to the question. — John Bollinger, Nov 12 '17 at 19:00
As for your problem, you say that your RAM size is 64 GiB, but is that the *physical* memory size, or the *virtual* memory size? — Some programmer dude, Nov 12 '17 at 19:00
Also, some system (like Windows IIRC) actually allow you to allocate *more* than the available virtual memory, as long as you don't use the pages you allocate. If you start writing to the pages and they can't be swapped in then you will start seeing failures. — Some programmer dude, Nov 12 '17 at 19:01
@JohnBollinger my system is intel xeon E5 with 64GB RAM. X64 Ubuntu OS — anirudh, Nov 12 '17 at 19:02

Basile Starynkevitch · Accepted Answer · 2017-11-12T19:09:01.837

5

You might want to disable memory overcommitment.

^{(I really dislike this feature)}

See this question.

You should code and test against malloc failure.

Read more about virtual address space. See also this.

edited Nov 12 '17 at 19:09

answered Nov 12 '17 at 19:03

Basile Starynkevitch

223,805
18
296
547

1

Thanks to your answer, I found a great page about overcommit on Linux. Just wanted to mention it here, for others who might face the same issue. https://www.etalabs.net/overcommit.html – anirudh Nov 13 '17 at 07:23

Why can I allocate arrays that are larger than my RAM?

1 Answers1