1

I need to allocate memory for 808704000 floats, which is something like 3085 MB. My computer has 32 GB of memory, and runs 64 bit Linux (CentOS 6.6). Every time I try to allocate the memory the malloc operation fails. I use g++ 4.4.7.

Can anyone explain why I cannot allocate the memory? Is it possible to somehow force the program to be compiled in 64 bit mode?

void AllocateMemory(float *& pointer, int size, void** pointers,
                    int& Npointers, nifti_image** niftiImages,
                    int Nimages, const char* variable)
{
    pointer = (float*)malloc(size);
    if (pointer != NULL)
    {
        pointers[Npointers] = (void*)pointer;
        Npointers++;
    }
    else
    {
        printf("Could not allocate host memory for variable %s !\n",
               variable);        
        FreeAllMemory(pointers, Npointers);
        FreeAllNiftiImages(niftiImages, Nimages);
        exit(EXIT_FAILURE);        
    }
}

ulimit -a prints:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256261
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
zwol
  • 135,547
  • 38
  • 252
  • 361
user2714031
  • 43
  • 1
  • 5
  • Use `strace` on your program. And a failing `malloc` is setting `errno`, so use `perror` in that case. BTW, in C++ it is better to use `new`. And show some source code here please. – Basile Starynkevitch Feb 16 '15 at 15:05
  • 8
    Are you sure your executable is being compiled for 64 bits? (64-bit systems can run 32-bit executables). What happens if you run `file ` on your executable? – psmears Feb 16 '15 at 15:05
  • 1
    Also, checkout http://stackoverflow.com/questions/3463207/how-big-can-a-malloc-be-in-c – Andrew White Feb 16 '15 at 15:07
  • http://uce.uniovi.es/tips/Programming/Cpp/forc32bitscompilation.html – drescherjm Feb 16 '15 at 15:09
  • file FirstLevelAnalysis FirstLevelAnalysis: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped – user2714031 Feb 16 '15 at 15:16
  • 1
    Can you show the output of `ulimit -a` in the shell? It shows limits for your programs, check data and stack values. – ott-- Feb 16 '15 at 15:21
  • There is rarely a significant advantage to having a block of contiguous memory 3 GB in size: it isn't as if cache coherency isn't already thrown out the window (if, say, it was clumped in 1 MB chunks, you'd be getting an extra cache miss only 0.0004% of the time compared to pure contiguous memory, even if the compiler did perfect fetch-ahead). If you are dealing with that much data, possibly a structure different than a flat array might be worth considering, or even a memory allocation system fancier than `malloc` if you really need linear addressing. – Yakk - Adam Nevraumont Feb 16 '15 at 15:25
  • @Yakk: In scientific computing, an array of that size (e.g. a big matrix) isn't that unusual. – Albert Feb 17 '15 at 07:59

1 Answers1

4

Make it size_t size. int is usually 32 bit, including the sign, thus the max number is 2^31 = 2,147,483,648 < sizeof(float) * 808,704,000 = 3,234,816,000.

Thus, int(sizeof(float) * 808704000) < 0 is an overflow.

Then, because malloc expects a size_t, it will sign-extended it to 64-bit, then reinterpreted as unsigned, giving huge number > 2^63. (Thanks ElderBug.)

Community
  • 1
  • 1
Albert
  • 65,406
  • 61
  • 242
  • 386
  • 1
    To add to this answer, `(int)3,234,816,000` is negative, and is sign-extended to 64-bit, then reinterpreted as unsigned, giving huge number > 2^63. – ElderBug Feb 16 '15 at 15:38