Why is file I/O in large chunks SLOWER than in small chunks?

Question

If you call ReadFile once with something like 32 MB as the size, it takes noticeably longer than if you read the equivalent number of bytes with a smaller chunk size, like 32 KB.

Why?

(No, my disk is not busy.)

---

Edit 1:

Forgot to mention -- I'm doing this with FILE_FLAG_NO_BUFFERING!

---

Edit 2:

Weird...

I don't have access to my old machine anymore (PATA), but when I tested it there, it took around 2 times as long, sometimes more. On my new machine (SATA), I'm only getting a ~25% difference.

Here's a piece of code to test:

#include <memory.h>
#include <windows.h>
#include <tchar.h>
#include <stdio.h>

int main()
{
    HANDLE hFile = CreateFile(_T("\\\\.\\C:"), GENERIC_READ,
        FILE_SHARE_READ | FILE_SHARE_WRITE, NULL,
        OPEN_EXISTING, FILE_FLAG_NO_BUFFERING /*(redundant)*/, NULL);
    __try
    {
        const size_t chunkSize = 64 * 1024;
        const size_t bufferSize = 32 * 1024 * 1024;
        void *pBuffer = malloc(bufferSize);

        DWORD start = GetTickCount();
        ULONGLONG totalRead = 0;
        OVERLAPPED overlapped = { 0 };
        DWORD nr = 0;
        ReadFile(hFile, pBuffer, bufferSize, &nr, &overlapped);
        totalRead += nr;
        _tprintf(_T("Large read: %d for %d bytes\n"),
            GetTickCount() - start, totalRead);

        totalRead = 0;
        start = GetTickCount();
        overlapped.Offset = 0;
        for (size_t j = 0; j < bufferSize / chunkSize; j++)
        {
            DWORD nr = 0;
            ReadFile(hFile, pBuffer, chunkSize, &nr, &overlapped);
            totalRead += nr;
            overlapped.Offset += chunkSize;
        }
        _tprintf(_T("Small reads: %d for %d bytes\n"),
            GetTickCount() - start, totalRead);
        fflush(stdout);
    }
    __finally { CloseHandle(hFile); }

    return 0;
}

Result:

Large read: 1076 for 67108864 bytes
Small reads: 842 for 67108864 bytes

Any ideas?

Answer 1

This is not specific to windows. I did some tests a while back with the C++ iostream library and found there was an optimum buffer size for reads, above which performance degraded. Unfortunately, I no longer have the tests, and I can't remember what the size was :-). As to why, well there are a lot of issues, such as a large buffer possibly causing paging in other applications running at the same time (as the buffer can't be paged).

Answer 2

When you perform the 1024 * 32KB reads are you reading into the same memory block over and over, or are you allocating a total of 32MB to rad into as well and filling the entire 32MB?

If you're reading the smaller reads into the same 32K block of memory, then the time difference is probably simply that Windows doesn't have to scavenge up the additional memory.

---

Update based on the FILE_FLAG_NO_BUFFERING addition to the question:

I'm not 100% certain, but I believe that when FILE_FLAG_NO_BUFFERING is used, Windows will lock the buffer into physical memory so it can allow the device driver to deal with physical addresses (such as to DMA directly into the buffer). It could (I believe) do this by breaking up a large request into smaller requests, but I suspect that Microsoft might have the philosophy that "if you ask for FILE_FLAG_NO_BUFFERING then we assume you know what you're doing and we're not going to get in your way".

Of course locking 32MB all at once instead of 32KB at a time will require more resources. So this would be kind of like my initial guess, but at the physical memory level rather than the virtual memory level.

However, since I don't work for MS and don't have access to Windows source, I'm going by vague recollection from times when I worked closer with the Windows kernel and device driver model (so this is more or less speculation).

Answer 3

Your test is including the time it take to read in file metadata, specifically, the mapping of file data to disk. If you close the file handle and re-open it, you should get similar timings for each. I tested this locally to make sure.

The effect is probably more severe with heavy fragmentation, as you have to read in more file to disk mappings.

EDIT: To be clear, I ran this change locally, and saw nearly identical times with large and small reads. Reusing the same file handle, I saw similar timings from the original question.

Answer 4

when you have done FILE_FLAG_NO_BUFFERING that means that the operating system will not buffer the I/O. So each time you call the read function it will make a system call which will fetch each time the data from the disk. Then to read one file with a fixed size if you use less buffer size then more system calls are needed so more user space to kernel space and for each time a disk I/O is initiated. Instead if you use larger block size then for the same file size to be read there would be less system calls required so the user to kernel space switches would be lesser, and the number of times the disk i/O initiated will also be lesser. This is why, generally larger block will require less time to read.

Try reading the file only 1 byte at a time without buffering, and try with 4096bytes block then and see the difference.

Answer 5

A possible explanation in my opinion would be command queueing with FILE_FLAG_NO_BUFFERING, since this does direct DMA reads at low level.

A single large request will of course still necessarily be broken into sub-requests, but those will likely be sent more or less one after another (because the driver needs to lock the pages and will in all likelihood be reluctant to lock several megabytes lest it hits the quota).

On the other hand, if you throw a dozen or two dozen requests at the driver, it will just forward them to the disk and the disk and take advantage of NCQ.

Well, that's what I'm thinking might be the reason anyway (this does not explain why the exact same phenomenon happens with buffered reads though, as in the Q that I linked to above).

Answer 6

What you are probably observing is that when using smaller blocks, the second block of data can be read while the first is being processed, then the third read while the second is being processed, etc. so that the speed limit is the slower of the physical read time or the processing time. If it takes the same amount of time to process one block as to read the next, the speed could be double what it would be if processing and reading were separate. When using larger blocks, the amount of data that is read while the first block is being processed will be limited to amount smaller than the block size. When the code is ready for the next block of data, part of it will have been read but some of it will not; it will thus be necessary for the code to wait while the remainder of the data is fetched.