Error allocating memory of unsigned longs in C

Question

I am writing a C program that needs to use some pretty large arrays. I allocate memory for my array by calling:

unsigned long *visited = declareArrayofLongs(my_pow(2, d));

The declareArrayofLongs() function is:

unsigned long int* declareArrayofLongs(unsigned long int length)
{
    unsigned long int *myarray = (unsigned long int*) malloc(length*sizeof(unsigned long int));
    if (myarray==NULL)
        printf("Error allocating memory for unsigned longs!\n");
    unsigned long int i;
    for(i=0; i<length; i++)
        myarray[i] = 0;
    return myarray;
}

And my_pow() is

unsigned long int my_pow(int base, int exp)
{
    unsigned long int pow=1;
    int i=0;
    for(i=0; i<exp; i++){
        pow = (unsigned long int)base*pow;
    }
    return pow;
}

When the program runs and gets up to around d=29, I get "Error allocating memory for unsigned longs!". I have 16gb of RAM on my system. Shouldn't I be able to handle this amount of allocation?

Only one visited array is declared at a time. I have a free(visited) call before re-allocating.

That's not depends on how much HDD you are you are using. That's depends on how much heap memory OS allocates for you. — Avinash, Dec 22 '14 at 15:32
If you look into your map file or into the linker script, it can help you understand how much heap memory is available for your program. It is really not depending on your physical memory. — Eugene Sh., Dec 22 '14 at 15:34
Though not related to your problem, don't cast the return value of `malloc()`. See http://stackoverflow.com/q/605845/3488231 — user12205, Dec 22 '14 at 15:35
1) What is the `sizeof(unsigned long)`: 4, 8, ? 2) What is the value of `SIZE_MAX`? — chux - Reinstate Monica, Dec 22 '14 at 15:38
@chux, `sizeof(unsigned long)` is 4. I'm having trouble printing out `SIZE_MAX` though. `printf("%d", SIZE_MAX);` prints out -1? — theQman, Dec 22 '14 at 15:44
BTW, statically allocating this array (to the largest possible size that you need) in the data section, i.e., global and/or `static unsigned long int myarray[1<<29];`, might solve the issue (with the trade-off of a very large executable file). — barak manos, Dec 22 '14 at 15:47
@barakmanos That's a possibility, but I doubt it will work, as 2^29 4-byte integers requires 2^31 bytes of virtual address space, which is larger than will fit in most 32-bit process images, no matter how it's requested. — twalberg, Dec 22 '14 at 15:50
@twalberg: Does the entire executable image have to be in memory when the process is being executed? I'm quite (though not entirely) sure that it doesn't. — barak manos, Dec 22 '14 at 15:54
@theGman When printing the value of `SIZE_MAX`, use the matching print specifier of `"%zu"`. — chux - Reinstate Monica, Dec 22 '14 at 15:54
Try `unsigned long int *myarray = calloc(length, sizeof *myarray);` [See #4](http://stackoverflow.com/questions/27597725/what-are-the-three-conditions-need-to-checked-before-re-allocating-memory-dynami/27598539#27598539) — chux - Reinstate Monica, Dec 22 '14 at 15:57
@barakmanos No, the entire image does not have to be in (physical) memory at any particular point, but when you allocate memory in C code, there is a portion of the virtual memory address space that has a mapping created for it, so it can be demand-paged as necessary. In this case, I believe that, once the mapping for the executable itself, and the kernel portions, and the shared libraries are all accounted for, there is no single virtual address range left (out of the maximum 2^32 bytes) that can accommodate 2^31 bytes, thus the allocation will fail. — twalberg, Dec 22 '14 at 15:57
@twalberg: Especially not with a 16GB of RAM, as mentioned in the question. Thanks :) — barak manos, Dec 22 '14 at 15:59
@barakmanos Having 16GB of RAM on a 32-bit system only means that you can successfully run a lot more processes at once. Any single process is going to be limited to 2^32 (=4GB) bytes of RAM (virtual or physical) because it only has a 32-bit address space. Less than that, actually, because the kernel takes up some (typically 1GB on Linux), the executable itself and any shared libraries take up more, data, stack, etc. all use portions of it as well. — twalberg, Dec 22 '14 at 16:01
Although OP's address space likely is limited to a 32-bit address space in this case, a 32-bit system (32-bit being the processor native size), may use pointers that exceed a 32-bit address space. Processor bit width and memory address space are separate issues. — chux - Reinstate Monica, Dec 22 '14 at 16:03
@barakmanos Why would one get a huge executable file in this case? — glglgl, Dec 22 '14 at 16:13
@glglgl Poor compilers put the entire data space in the generated executables - even large zero filled arrays. Good compilers identified large array that need to be zero-filled and only have a small increase to the executable size to support that. — chux - Reinstate Monica, Dec 22 '14 at 16:17
SIZE_MAX is 4294967295. Why is this relevant? Also, I'm running cygwin on a 64 bit windows system, but I think cygwin might be the 32 bit version. I'm compiling with gcc. — theQman, Dec 22 '14 at 16:17
@chux Ouch. Such compilers really exist (outside of devel labs)? I didn't know that... — glglgl, Dec 22 '14 at 16:19
@chux: As far as I'm aware of, the data-section of the executable image has to be pre-allocated to the full size required. I don't see how that has anything to do with whether or not the global data (the array in this case) is initialized to zeros or to any other value (or even left uninitialized). In other words, every global and/or `static` variable "automatically" increases the size of the executable image. — barak manos, Dec 22 '14 at 16:19
`SIZE_MAX` gives use the true allowable range of `size_t`. That is the type used in `malloc(size_t)`. That is the first step in solving this. That does not mean code can allocate 4294967295 bytes, but that is certainly an limiting factor. — chux - Reinstate Monica, Dec 22 '14 at 16:20
@glglgl It has happened from time-to-time, usually as you suggest, on early developing compiler for a new platform. Quite astonishing to see the executable size 1GB when 1MB is expected. — chux - Reinstate Monica, Dec 22 '14 at 16:23
@barakmanos You should be aware about the difference between the `.data` section (which is preallocated with data in the binary) and the `.bss` section (with is 0-initialized). This difference exists in most compilers (except those in a very early developing stage). — glglgl, Dec 22 '14 at 16:25
@barak manos The format of an executable is not specified by the C spec, many layouts exist. If code had a global `int a[1000000000];`, that memory needs to be zero'd at run time. An executable could have a segment identified that is small and the loader insures the large zero'd space. — chux - Reinstate Monica, Dec 22 '14 at 16:26
@glglgl: I'm aware of the data-section divided into initialized and uninitialized data, but regardless of that, since the (virtual) address of every global variable is constant throughout the execution of the program - doesn't that mean that this variable has to be "pre-allocated" at a constant memory address (which would impose the pre-allocation of the entire data-section)? — barak manos, Dec 22 '14 at 16:30
@theGman I _think_ your problem is simply that the compiler enforces an allocation limit well less the `SIZE_MAX`. I would not be surprised if `malloc(((1lu << 29) - 1) * sizeof (long))` worked. If the pointer size > 32-bit, the `calloc()` idea, may work. Else use a different compiler mode/different compiler. — chux - Reinstate Monica, Dec 22 '14 at 16:33
@barak manos If you idea is true or not, it still does not mean that the executables needs to carry a mirror image of a large zero filled array. The executable is rarely directly loaded in memory. The loader can create the large zero memory space based on info in the executable. Maybe the entire executable is a mirror image of what is to be loader, but the file is also compressed. Lots of approaches to reduce size. — chux - Reinstate Monica, Dec 22 '14 at 16:37

score 1 · Answer 1 · answered Dec 22 '14 at 15:32

1

You are not showing my_pow. Since your code doesn't work, what makes you think it does what you think it does?

You may be running 32 bit code, in which case you won't get more than about 3GB, no matter how much RAM.

answered Dec 22 '14 at 15:32

gnasher729

51,477
5
75
98

I have added the my_pow function to my originial post. The code works for smaller values of `d`, it's only once d gets large (around 29) that I get the allocation error. – theQman Dec 22 '14 at 15:39
1

@theGman If this is a 32-bit system (I'm guessing it is), and `unsigned long` is 4 bytes, then you're asking for 2^31 bytes of memory in your `malloc()` call when `d == 29`. That's probably more than the C library will grant, although since you don't mention what platform and compiler you're using, we can't really say anything about the specific limits... – twalberg Dec 22 '14 at 15:49
@twalberg, I'm running cygwin on a 64 bit windows system, but I think cygwin might be the 32 bit version. I'm compiling with gcc. – theQman Dec 22 '14 at 16:05
@theGman In that case, and assuming your `gcc` is the one that came with Cygwin, and not another version loaded on top of it, it's probably only going to be able to compile 32-bit apps, which most likely cannot support allocation requests of 2^31 bytes (see comments under question for more detail). You can verify the compiler target architecture by running `gcc -v`. If it says `Target: i?86-something` (maybe `x86_32` as well - I don't have one handy to verify) it's a 32-bit compiler. If it says `Target: x86_64-something` it's 64-bit. You could try installing the 64-bit Cygwin... – twalberg Dec 22 '14 at 16:11
@iharob The last sentence of this post is actually the right answer, IMHO. – glglgl Dec 22 '14 at 16:18

Error allocating memory of unsigned longs in C

1 Answers1