Memory allocation and process memory use

Question

I would like to understand why data dynamically allocated called multiple times uses so much memory than the one directly specified on code or allocated with a single call of malloc.

Examples

As example, I made the following two codes, in C:

test1.c: int x is allocated with malloc

int main (void)
{
    int *x;
    int i, n=1048576; //n=1024*1024;

    printf("size = %lu\n", n* sizeof(int));

    for(i=0; i<n; i++)
    {
        x = malloc(sizeof(int));
        *x=i;
    }
    printf("Look at top and then press something to finish.");fflush(stdout);
    getc(stdin);
    return 0;
}

I did not use free here to keep it simple. When the program is waiting for interaction, I look at the top function in another terminal, and it shows me this:

PID  USER   PR  NI  VIRT   RES    SHR  S  %CPU %MEM  TIME+   COMMAND                                    
1384 root   20  0   41300  34076  1300 S  0.0  3.3   0:00.47 test1

test2.c: int x is not allocated dynamically

int main (void)
{
    int x[1048576]; //x[1024*1024]
    int i, n=1048576;

    printf("size = %lu\n", n* sizeof(int));

    for(i=0; i<n; i++)
    {
        x[i]=i;
    }
    printf("Look at top and then press something to finish.");fflush(stdout);
    getc(stdin);
    return 0;
}

And top shows me:

PID  USER   PR  NI  VIRT    RES    SHR  S  %CPU %MEM  TIME+   COMMAND                                    
1352 root   20  0   12404   5500   1304 S  0.0  0.5   0:00.05 test2

I also did a third code, which has the same result that test2, where I used:

x = malloc(n*sizeof(int));
for(i=0; i<n; i++)
    {
        x[i]=i;
    }

Why so much difference on the memory use of the processes? That is because malloc request new memory pages and are there memory being wasted? Or malloc allocates more memory?

test1 uses 3.3% of the total memory and test2 uses 0.5%.

Environment:

I am executing those tests on Centos 5 64 bits inside docker.

Memory on the virtual environment:

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            995          98         845           3          51         808
Swap:          1162         194         967

Possible duplicate of [Malloc vs custom allocator: Malloc has a lot of overhead. Why?](http://stackoverflow.com/questions/13064850/malloc-vs-custom-allocator-malloc-has-a-lot-of-overhead-why) — Ken Y-N, Jul 21 '16 at 01:02
Take a look at this answer: http://stackoverflow.com/questions/10540845/linux-heap-structure-and-the-behaviour-with-malloc-and-free — JurekM, Jul 21 '16 at 01:08
This is related to 'malloc vs custom allocator', but not quite the same as that. — Jonathan Leffler, Jul 21 '16 at 01:09
This is also related to SO 1054085 (Linux heap structure and behaviour of malloc and free), but not quite the same. (Or maybe I'm just being unusually fussy today — not sure.) — Jonathan Leffler, Jul 21 '16 at 01:16
Note that test2 is allocating space from the stack which is pre-allocated at program load and start up time. The third example does a one time allocation from the heap, so it should use more memory than the second example. Getting back to test2, a 4 megabyte+ stack seems unusually large. I'm wondering if a smart compiler would detect that x[] is local to main and uses a static like variable (from program memory instead of stack). — rcgldr, Jul 21 '16 at 01:30
@rcgldr: the overhead of 16 bytes or so on a 4 MiB heap allocation is negligible, whereas the overhead of 4-12 bytes (possibly 4-28 bytes overhead) on each of a million or so allocations of 4 bytes is rather significant. Also, a million separate allocations are not readily manageable: you need an array of a million pointers to hold the million separate values, which leads to messy freeing, etc. — Jonathan Leffler, Jul 21 '16 at 04:10
@JonathanLeffler - my point was that the memory used in test 2 would be the same regardless of the size of x[], since it's allocated from the stack, which in turn is preallocated at load / startup. Test 3 should also allocate the same amount of space for the stack, and then in addition, allocate 4 MB of ram from the heap. — rcgldr, Jul 21 '16 at 07:17

score 6 · Accepted Answer · answered Jul 21 '16 at 01:04

6

Every memory allocation has to be tracked so that free() can release the space for reuse. In practice, that means there's a minimum memory size that's allocated; it can be 8 or 16 bytes for a 32-bit program and 16-32 bytes for a 64-bit program (it depends on the system and the version of the C library in use).

When you allocate the one million integers separately, each one of them uses 8-32 bytes, so you've actually used 8-32 MiB of memory. When you allocate the one million integers in a single array on the stack, you're using 4 MiB of memory.

Hence you see a substantial difference in the process size.

The first program leaks almost all the memory, of course, but that's tangential to the question you're asking.

answered Jul 21 '16 at 01:04

Jonathan Leffler

730,956
141
904
1,278

1

Jonathan nice answer! However, notice that the last sentence might not be needed, since OP mentioned that he deliberately not included the `free()` for simplicity. :) – gsamaras Jul 21 '16 at 01:10
1

Yes; my last sentence isn't 100% necessary, but it isn't harmful either. The simplicity gained by leaving `free` out of the question is significant; the code would need 'a million' pointers to store the million allocated integers, adding to the memory overhead (4 or 8 MiB more needed). – Jonathan Leffler Jul 21 '16 at 01:12

score 2 · Answer 2 · answered Jul 21 '16 at 01:03

The system needs to do some housekeeping every time a user asks for some memory. That's should come in mind when just calling free with just your pointer suffices for the system to de-allocate its memory.

So, in your first example, you request memory n times, while on the second just once. The memory that you intend to use is the same, but the information the system has to "remember" is not.

Memory allocation and process memory use

Examples

Environment:

2 Answers2