Uploading file to server throws out of memory exception

Question

I am trying to implement a file upload system with asp.net web api and I am running into a problem. I am trying to get the multipart form data into a memory stream so it can be written to either disk or blob storage depending on the service layer implementation. The problem is it works fine for small files but I am trying to upload a file of 291 MB and it is throwing an out of memory exception. Here is the code:

if (!Request.Content.IsMimeMultipartContent())
{
    Request.CreateErrorResponse(HttpStatusCode.UnsupportedMediaType, "Request must be multipart.");
}

var provider = new MultipartMemoryStreamProvider();

try
{
    await Request.Content.ReadAsMultipartAsync(provider);

    var infoPart = provider.Contents.Where(x => x.Headers.ContentDisposition.Name.Replace("\"", string.Empty) == "fileInfo").SingleOrDefault();
    var filePart = provider.Contents.Where(x => x.Headers.ContentDisposition.Name.Replace("\"", string.Empty) == "filePart" && x.Headers.ContentDisposition.FileName != null).Single();
    byte[] file = null;

    using (Stream stream = filePart.ReadAsStreamAsync().Result)
    {
        using (MemoryStream memory = new MemoryStream())
        {
            stream.CopyTo(memory);
            file = memory.ToArray();
        }
    }

    string fileContentType = filePart.Headers.ContentType.MediaType;

    FileDto result = _fileService.AddFileToResource(Variables);
    string uri = Url.Link("DefaultGet", new { id = result.ID });
    return Request.CreateResponse(HttpStatusCode.OK);

The part that throws the error is on the

await Request.Content.ReadAsMultipartAsync(provider);

The exact error is

Error writing MIME multipart body part to output stream.

with inner exception of

Exception of type 'System.OutOfMemoryException' was thrown.

I have tried creating a custom BufferPolicySelector as shown in the second answer of this post and many other places but that doesn't seem to help at all.

I have also added to my web.config:

<httpRuntime targetFramework="4.5" maxRequestLength="307200"/>

and

<security>
  <requestFiltering>
    <requestLimits maxAllowedContentLength="367001600"/>
  </requestFiltering>
</security>

I could be wrong here but the way I read your code you will have 3 copies of this file in memory in this one snippet: the request content stream, the memory stream and then the byte array file. Maybe the reuqest result stream will truly stream but both 'memory' and 'file' will be in RAM concurrently. — dkackman, Apr 17 '14 at 19:52
Thanks for the advice. How would I improve this? It doesn't really help my issue because it throws the error when it's reading the content in await Request.Content.ReadAsMultipartAsync(provider); but I would definitely like to understand how to make it as efficient as possible — GBreen12, Apr 17 '14 at 19:57
verify also that maxRequestLength should be greater or equal to value assigned to RequestLengthDiskThreshold — fnostro, Apr 17 '14 at 20:02
@gmoney12 would be hard to refactor this for you as it is difficult to tell what exactly you are doing in there. 'stream' and 'file' don't seem to be used so as far as I can see that entire block can be deleted. — dkackman, Apr 17 '14 at 20:14
You should not be doing `using (Stream stream = filePart.ReadAsStreamAsync().Result)` you need to do `using (Stream stream = await filePart.ReadAsStreamAsync())` instead, you could end up deadlocking your server doing the first way. You should (almost) never have a `.Result` or a `.Wait()` in a function that is declared `async`. — Scott Chamberlain, Apr 17 '14 at 22:11

Ashar · Accepted Answer · 2020-02-03T14:52:55.860

One solution would be to use MultipartFormDataStreamProvider instead of the MultipartMemoryStreamProvider to avoid the out of memory exception during the call

Request.Content.ReadAsMultipartAsync(..)

I was facing a similar problem while trying to use a MemoryStreamProvider while reading the MultiPart file contents for a large file (> 100 MB). The work around that worked for me was to use MultipartFormDataStreamProvider. The file is written to the disk during the ReadAsMultipartAsync call and can be later loaded back in if you need it in memory.

Here is an example taken from:

Sending HTML Form Data in Web API: File Upload and Multipart MIME

    string root = HttpContext.Current.Server.MapPath("~/App_Data");
    var provider = new MultipartFormDataStreamProvider(root);

    try
    {
        // Read the form data.
        await Request.Content.ReadAsMultipartAsync(provider);

        // This illustrates how to get the file names.
        foreach (MultipartFileData file in provider.FileData)
        {
            Trace.WriteLine(file.Headers.ContentDisposition.FileName);
            Trace.WriteLine("Server file path: " + file.LocalFileName);
        }
        return Request.CreateResponse(HttpStatusCode.OK);
    }
    catch(...)

This is what I ended up doing. Not sure if it is the best way but I never could figure out how to get it to work the other way — GBreen12, Mar 18 '15 at 16:22
Is there a way to do this in asp.net core? I only have Request.Body not Request.Content and there is no ReadAsMultipartAsync method. — Christopher Painter, Jul 30 '21 at 16:35

fnostro · Answer 2 · 2014-04-17T20:52:02.823

0

check your webconfig for file upload limits

<system.webServer>
        <security>
            <requestFiltering>
                <requestLimits maxAllowedContentLength="524288000"/>
            </requestFiltering>
        </security>
</system.webServer>

From OP Edit:

<httpRuntime targetFramework="4.5" maxRequestLength="307200"/>

The maxRequestLength is in Kb - this is the buffering threshold, you are requesting a 300MB buffer - probably not good. set it lower

edited Apr 17 '14 at 20:52

answered Apr 17 '14 at 19:46

fnostro

4,531
1
15
23

1

I already have that in the web.config I'll edit to add that information – GBreen12 Apr 17 '14 at 19:54
Keep in mind that thevalue above is in bytes with a max value of UINT, so about 4GB. The value above is about 500MB – fnostro Apr 17 '14 at 19:59
Right, but I'm only uploading a 290 MB file so it should upload fine right? Or am I thinking about it wrong? – GBreen12 Apr 17 '14 at 20:00
Well that's why I don't understand why this isn't working because I have a 300 MB max content length and I even tried putting it to 500 MB like you have but it still throws memory out of bounds – GBreen12 Apr 17 '14 at 20:10
This is just a setting of content size limit for the request, but your problem is that your process run out of memory – Matt Apr 17 '14 at 20:19
it sounds like it's trying to stuff the entire file into a memory stream - maybe look to make sure the upload file buffer is being flushed properly – fnostro Apr 17 '14 at 20:32

score -1 · Answer 3 · answered Dec 04 '14 at 15:20

-1

Verify the permissions on the folder on the server.

Or

Put httpRuntime maxRequestLength in the web.config for example:

<httpRuntime maxRequestLength="5242880" />

answered Dec 04 '14 at 15:20

Erick Martinez

283
4
7

Uploading file to server throws out of memory exception

3 Answers3