How to malloc a block size of UINT_MAX?

Question

I want to create an array with exactly UINT_MAX number of indexes. This has been extremely hard for some reason. I've tried the following:

• char arr[UINT_MAX]; // Compiler claims array size cannot be negative
• char* const arr = calloc(UINT_MAX, sizeof(char)); // Runs but seg fault when accessed
• char* const arr = malloc(sizeof(char) * UINT_MAX); // arr is NULL

I don't understand what's going on. Is it my HEAP size too low? If so, how can I increase it? Can malloc/calloc not handle blocks of that nature? I've found nothing useful in the malloc or calloc API pages.

This question applies to both C and C++.

Windows 7 64 Bit, 16GB RAM, CMake 3.6.1, GCC 7.11.1, and compiled on CLion 64 bit.

Answer 1

First, let me point out is that you can and should use errno to check why API calls fail.

Now, Let's examine your tries:

stack allocation - char arr[UINT_MAX]

You are trying to allocate 4294967295 bytes on your stack, which is not feasible, since operating systems limit the stack size to much smaller sizes. You could try and manipulate it using the APIs specified in this manual.

malloc

A heap allocation will fail in most cases for a single reason: The operating system couldn't allocate a big enough contiguous memory.

It's not that your system doesn't have 4GB of free memory, but it is entirely possible that there no single chunk of contiguous memory that suits your request. So, just like the malloc's MSDN page points out:

malloc returns a void pointer to the allocated space, or NULL if there is insufficient memory available

calloc

malloc and calloc use the same underlying mechanisms, so if one fails, you should expect the second one to fail as well. See remarks section of calloc's MSDN page

---

Interesting facts about malloc on Linux platforms

Following requests from the comments, I will point out some interesting facts about malloc's implementation on Linux, since it differs significantly!

First of all, Linux specifically specifies it uses an optimistic memory management algorithm:

By default, Linux follows an optimistic memory allocation strategy. This means that when malloc() returns non-NULL there is no guarantee that the memory really is available. In case it turns out that the system is out of memory, one or more processes will be killed by the OOM killer

This means that the memory is not actually owned and dedicated for the callee, but rather the operating system hopes that the memory will be available for him when he needs it, and if it isn't, it reverts to some nasty means.

Edit: As @Basile Starynkevitch correctly pointed out, this mechanism depends on the memory overcommitment switch, which is enabled by default, but can be disabled.

Answer 2

It is operating system specific. But not checking against failure of malloc or calloc leads to undefined behavior (e.g. your segfault with calloc) and is always wrong (and a basic mistake you should avoid doing). Remember that malloc and calloc can fail, and are more likely to fail when you request or have already used a lot of memory (because virtual memory is a limited resource). Notice that sizeof(char) is always 1.

The following program

 // file hatefiend.c
 #include <stdio.h>
 #include <stdlib.h>
 #include <unistd.h>
 #include <limits.h>

 int main(int argc, char**argv) {
   size_t sz = (argc>1)?(size_t)(atol(argv[1])):UINT_MAX;
   char *ad = malloc(sz);
   if (!ad) {perror("malloc"); exit(EXIT_FAILURE); };
   for (unsigned long ix=0; ix<(unsigned long)sz; ix += 1024)
     ad[ix] = (char)(' ' + (ix & 0x3f));
   printf ("ad=%p sz=%zd\n", ad, sz);
   /// the code below is optional and Linux specific, 
   /// remove it or adapt it for Windows
   char cmd[80];
   snprintf(cmd, sizeof(cmd), "pmap %d", (int)getpid());
   printf ("command: %s\n", cmd);
   fflush(NULL);
   int bad = system(cmd);
   if (bad) return  EXIT_FAILURE;
   return 0;
 }   

works well on my Linux/Debian/x86-64 laptop system with 16 Gbytes RAM (and won't be optimized by gcc -Wall -O hatefiend.c -o hatefiend removing ad because it is used in printf and filled) giving:

% ./hatefiend
ad=0x7fc3d5c09010 sz=4294967295
command: pmap 31644
31644:   ./hatefiend
000056355b8e3000      4K r-x-- hatefiend
000056355bae3000      4K r---- hatefiend
000056355bae4000      4K rw--- hatefiend
000056355c3de000    132K rw---   [ anon ]
00007fc3d5c09000 4194308K rw---   [ anon ]
00007fc4d5c0a000   1612K r-x-- libc-2.24.so
00007fc4d5d9d000   2048K ----- libc-2.24.so
00007fc4d5f9d000     16K r---- libc-2.24.so
00007fc4d5fa1000      8K rw--- libc-2.24.so
00007fc4d5fa3000     16K rw---   [ anon ]
00007fc4d5fa7000    140K r-x-- ld-2.24.so
00007fc4d6190000      8K rw---   [ anon ]
00007fc4d61c7000     12K rw---   [ anon ]
00007fc4d61ca000      4K r---- ld-2.24.so
00007fc4d61cb000      4K rw--- ld-2.24.so
00007fc4d61cc000      4K rw---   [ anon ]
00007ffe8f561000    132K rw---   [ stack ]
00007ffe8f5b7000     12K r----   [ anon ]
00007ffe8f5ba000      8K r-x--   [ anon ]
 total          4198476K

as you can see from the output of pmap(1) the memory is indeed allocated.

and you could easily adapt it for your Windows system: remove <unistd.h> and change the snprintf to build the command showing the virtual address space, i.e. replace pmap by some Windows specific utility; or just remove all the building of the command i.e. cmd and its usage...

So do yourself a favor: install and use Linux (that could be seen as a provocation or as a useful hint). It is likely that on Windows your malloc or calloc has failed. Or work hard to understand why did that happen, and perhaps configure something in your system to let a huge malloc succeeds. How to do that is a sysadmin issue, out of scope on Stack Overflow (but you could ask on SuperUser). Beware of memory overcommitment (I usually disable that).

Is it my HEAP size too low? If so, how can I increase it?

You probably want to increase the virtual memory or the virtual address space limit (since heap memory is what malloc gives). How to do that is a sysadmin issue (should be asked on SuperUser), and you'll need to give a lot more details to get help.

In a comment you mentioned that your size_t is a 32 bits integer (which surprises me a lot; are you compiling in 64 bits mode? on my Linux/x86-64 machine it is 64 bits, like some uint64_t or unsigned long). Then I believe that what you want is impossible on your system (because malloc implementation would need some extra space -e.g. a few extra words for book keeping- and perhaps counts that overhead in some internal size_t ....)

---

Remember that this malloc (not very useful, but very quick) is following the letter of the C standard.

---

However I do believe that you have some XY problem, and you could achieve your goals (which are unstated) otherwise. This is why motivating questions is important on Stack Overflow. Maybe declaring arr as a static variable might help: static char arr[UINT_MAX];.... but read also this for the disadvantages.

Answer 3

Not sure why you need that much char array, but anyway

char arr[UINT_MAX];

That's too huge for stack size.. No compilers allocate this much stack size. you can increase the stack size while compiling with flag -stack but its still limited to the your system configurations.

you can check your limits by invoking command ulimit -s and set it to unlimited ulimit -s unlimited before trying that..However it can still fail.

malloc returns NULL as it can't allocate the memory you asked for and same for calloc.

Increase you virtual memory or physical memory or both to get this memory. Even if memory is available, it may well be fragmented to a extent its not possible for system to honour your request.