What is maximum size when read binary files?

Question

Let say I want to read array of doubles. Then I use this code:

FILE *fp;
int n=100;
fp=fopen("file.bin","rb");
double *array=(double*)calloc(n,sizeof(double));
fread(array,sizeof(double),n,fp);
fclose(fp);

What is the maximum n that I can safely use for fread() function?

That depends on your machine. A safer way would be to measure the length of the file, e.g. using stat, or fseek to the end, ftell to get the length and fseek back to the start. (In your example you also need to capture the fread output to know how many doubles you actually read into the buffer.) But if you know the number of doubles in the file and are just worried about allocating 800 bytes, I think that'd be safe always. (And, for completeness, you ought to check the file handle returned from fopen is not null too.) — Rup, Sep 03 '15 at 15:59
[don't cast the result of malloc in C](http://stackoverflow.com/q/605845/995714) — phuclv, Sep 03 '15 at 16:52

score 3 · Answer 1 · edited Sep 03 '15 at 17:27

In your code? MAX_INT, which is platform dependent. Usually 2,147,483,647. But that's not guaranteed.

You're further constrained by real-world hardware and the state of your machine. malloc can only go grab the RAM that's available. If it's not there, it just fails. If you've got 256K of RAM in your machine, it's not much. It's far less than 256K. If you've got NSA's servers and God's own memory pool... but it's chock-full of cat videos, malloc will fail at any value.

You could do some real analysis and determine how much memory you'll have free on a given system performing a given list of tasks. Or you can keep it a small percentage of the assumed hardware, and just sort of hope they don't run out. I'd say it's important to be cognizant that your memory allocations can fail, and the program needs to fail gracefully.

This sort of uncertainty is why dynamic memory is a no-go for critical systems.

*If you've got NSA's servers and god's own memory pool...* - Nice emphasising. +1, — Am_I_Helpful, Sep 03 '15 at 16:17

score 2 · Accepted Answer · edited May 23 '17 at 10:26

The answer is provided in the C standard, 7.19.8. I'll use the N1256 draft.

size_t fread(void * restrict ptr,
          size_t size, size_t nmemb,
          FILE * restrict stream);

Since [7.19.8.1] provides no limitations, the values of size and nmemb can be up to the maximums provided by their type - SIZE_MAX [7.18.3.2], as long as the storage pointed to by ptr has sufficient size.

You're not guaranteed that fread actually will read that many elements:

7.19.8.1.2 The fread function reads, into the array pointed to by ptr, up to nmemb elements whose size is specified by size [...] If a partial element is read, its value is indeterminate. (emphasis mine)

As long as malloc can allocate it, you can fill it up with fread, although fread is free to read a smaller number of elements. For example, a conformant fread implementation could be reasonably limited to reading min(SIZE_MAX/size, nmemb) elements in one go.

Because of that, your use of fread is wrong. You must keep reading until you're done reading what you needed to read, or an error has occurred, or you've reached the end of file. Never test for feof prior to reading!.

void processData(double * data, size_t count);

void readData(void) {
  FILE *file = fopen("file.bin","rb");
  size_t n = 100;

  if (file) {
    double * array = calloc(n, sizeof(double));
    double * head = array;
    if (!array) break;
    while (n) {
      size_t n_read = fread(head, sizeof(double), n, file);
      head += n_read;
      n -= n_read;
      if (feof(file) || ferror(file)) break;
    }
    processData(array, head-array);
    fclose(file);
  }
}

score 1 · Answer 3 · answered Sep 03 '15 at 17:09

1

Use size_t n = SIZE_MAX to attempt to allocate the largest code can handle.

Various systems pose other limits.

Many systems require n*sizeof *array <= SIZE_MAX for calloc()/fread(). So use n = SIZE_MAX/sizeof *array
Allocatable memory which may be more than real memory as calloc() does not use memory, just allocates it. Even fread() might not use all of the memory it is given as the file input is small. Yet counting on that is not robust code.
In the end, once code attempts to use its allocation, out-of-memory faults could then occur.

With regard to "safely use", code should check for NULL right after the calloc().

Instead, recommend reading chunks of data and reallocating into one big array if needed afterward.

answered Sep 03 '15 at 17:09

chux - Reinstate Monica

143,097
13
135
256

This doesn't answer the question now :( – Kuba hasn't forgotten Monica Sep 04 '15 at 14:57
@Kuba Ober If `calloc()` can provide it, `fread()` can certainly use it. The question becomes "What is maximum size when calling `calloc()`?" `calloc()` can over allocate real memory. Memory systems typical later perform memory/disk swap when memory is _used_. On systems with 64bit `size_t`, code could request an insane size & receive a valid pointer & pass that to `fread()`. `fread()` reads, say a file 10x physical memory size. Result: lots of disk thrashing & a successful (and slow) read. Max `n` limits: `size_t`, maybe `size_t/sizeof object`: real memory usage back by swap files, etc. – chux - Reinstate Monica Sep 04 '15 at 15:16

What is maximum size when read binary files?

3 Answers3