IOException reading a large file from a UNC path into a byte array using .NET

Question

I am using the following code to attempt to read a large file (280Mb) into a byte array from a UNC path

public void ReadWholeArray(string fileName, byte[] data)
{
    int offset = 0;
    int remaining = data.Length;

    log.Debug("ReadWholeArray");

    FileStream stream = new FileStream(fileName, FileMode.Open, FileAccess.Read);

    while (remaining > 0)
    {
        int read = stream.Read(data, offset, remaining);
        if (read <= 0)
            throw new EndOfStreamException
                (String.Format("End of stream reached with {0} bytes left to read", remaining));
        remaining -= read;
        offset += read;
    }
}

This is blowing up with the following error.

System.IO.IOException: Insufficient system resources exist to complete the requested

If I run this using a local path it works fine, in my test case the UNC path is actually pointing to the local box.

Any thoughts what is going on here ?

score 4 · Accepted Answer · edited May 23 '17 at 09:57

4

I suspect that something lower down is trying to read into another buffer, and reading all 280MB in one go is failing. There may be more buffers required in the network case than in the local case.

I would read about 64K at a time instead of trying to read the whole lot in one go. That's enough to avoid too much overhead from chunking, but will avoid huge buffers being required.

Personally I tend to just read to the end of the stream rather than assuming that the file size will stay constant. See this question for more information.

edited May 23 '17 at 09:57

Community

1
1

answered Dec 22 '08 at 12:07

Jon Skeet

1,421,763
867
9,128
9,194

Any particular reason you use 4 pages of memory for your buffers? – Robert Davis Mar 19 '10 at 19:11
@Robert Davis: The important size isn't *memory* pages but *IO* buffers. If you read so much that it requires multiple IO requests, that's a bit wasteful - if you read so little that it fits into one request with space to spare, that's wasteful the other way. My experience is that 64K often makes a good compromise, but it depends on exactly what's happening at the lower levels. – Jon Skeet Mar 19 '10 at 19:24

score 1 · Answer 2 · answered Mar 19 '10 at 19:18

Also, the code as written needs to put the FileStream into a using block. Failing to dispose of resources is a very possible reason for receiving "Insufficient system resources":

public void ReadWholeArray(string fileName, byte[] data)
{
    int offset = 0;
    int remaining = data.Length;

    log.Debug("ReadWholeArray");

    using(FileStream stream = new FileStream(fileName, FileMode.Open, FileAccess.Read))
    {
        while (remaining > 0)
        {
            int read = stream.Read(data, offset, remaining);
            if (read <= 0)
                throw new EndOfStreamException
                    (String.Format("End of stream reached with {0} bytes left to read", remaining));
            remaining -= read;
            offset += read;
        }
    }
}

+1. Wow, the OP is > 1 year old and you noticed this. The buffer size was probably the major problem but this would have become one. — Kevin Brock, Mar 19 '10 at 19:33
@Kevin: I did an edit of the question (because of an edit by Jon Seigal), and it jumped right out at me. Lesson hard-learned. — John Saunders, Mar 19 '10 at 19:40

Joel Lucsy · Answer 3 · 2008-12-22T19:09:40.843

0

It looks like the array wasn't created with sufficient size. How big of an array is allocated prior to being passed in? Or were you assuming the Read function would reallocate the data array as needed? It won't. Edit: Er, maybe not, I just noticed the exception you got. Not sure now.

edited Dec 22 '08 at 19:09

answered Dec 22 '08 at 19:03

Joel Lucsy

8,520
1
29
35

IOException reading a large file from a UNC path into a byte array using .NET

3 Answers3