Memory exceeding while reading from stream writing to another one

Question

I am reading data from one stream and writing to another. The file I want to copy is 1.4 GB of size. Why is the memory used by my program growing all the time while reading and writing. How can i prevent an out of memory exception?

//Write data from URL
HttpWebRequest webRequestWrite = (HttpWebRequest)WebRequest.Create(WriteUrl);

webRequestWrite.AllowReadStreamBuffering = false;
webRequestWrite.AllowWriteStreamBuffering = false;

webRequestWrite.Method = "PUT";
using (Stream responseStreamWrite = webRequest.GetRequestStream())
{

   int chunkSize = 4194304;
   byte[] buffer = new byte[chunkSize];
   int bytesRead = 0;
   int totalBytes = 0;

   //Read data from URL
   HttpWebRequest webRead = (HttpWebRequest)WebRequest.Create(ReadUrl);
   webRead.AllowReadStreamBuffering = false;
   webRead.AllowWriteStreamBuffering = false;
   using (WebResponse webResponseread = webRead.GetResponse())
   using (Stream responseStreamRead = webResponseread.GetResponseStream())
   {
      while ((bytesRead = responseStreamRead.Read(buffer, 0, chunkSize)) > 0)
      {
        totalBytes += bytesRead;
        responseStreamWrite.Write(buffer.Take(bytesRead).ToArray(), 0, bytesRead);
        responseStreamWrite.Flush();
        responseStreamRead.Flush();
      }
   }

   responseStreamWrite.Flush();
}

possible duplicate of [Reading large text files with streams in C#](http://stackoverflow.com/questions/2161895/reading-large-text-files-with-streams-in-c-sharp) — prashantsunkari, Sep 28 '14 at 00:40
Are you getting OOM exception or just unhappy about memory usage? Note that you should probably be using `Stream.CopyTo` or even `Stream.CopyToAsync` instead of inventing your own. — Alexei Levenkov, Sep 28 '14 at 00:49
Can you set `SendChunked` to true on the `webRequestWrite` request? You may need to set the ContentLength as well. — Mike Zboray, Sep 28 '14 at 00:52
@mike z SendChunked did the trick. Please post it as answer and I will accept it. Thx. The memory consumption is now at 11 Mb. You do not need to set the ContentLenght — Chris, Sep 28 '14 at 13:41

score 1 · Accepted Answer · answered Sep 28 '14 at 20:38

You can try setting SendChunked to true on the request that you are writing to (webRequestWrite). Otherwise, the request will internally buffered until the request stream is closed, so the entire request can be sent as a single message. The underlying resource needs to support this transfer mode and may require you to set the content length first. The internal buffering here is different from AllowWriteStreamBuffering which buffers data for resending in the case of redirection or authentication requests.

score 0 · Answer 2 · edited May 23 '17 at 12:27

0

Try replacing Stream with BufferedStream.

http://msdn.microsoft.com/en-us/library/system.io.bufferedstream.read(v=vs.110).aspx

Might be duplicate question:

Reading large text files with streams in C#

edited May 23 '17 at 12:27

Community

1
1

answered Sep 28 '14 at 00:42

prashantsunkari

959
7
21

Not sure how this answers the question: OP can't pick type of the stream at all (coming from the WebRequest) and already reading in huge chunks. – Alexei Levenkov Sep 28 '14 at 00:47
Where are you seeing the out of memory exception. Is it at this step bytesRead = responseStreamRead.Read(buffer, 0, chunkSize) ? – prashantsunkari Sep 28 '14 at 00:51

Memory exceeding while reading from stream writing to another one

2 Answers2