1

Given two external APIs with one offering a download API and the other offering an upload API, both only allow using a Stream for data exchange, I need a way to efficiently transfer the data directly from download straight to upload.

Assume the APIs are as simple as:

Download (string url, Stream stream)

Upload (string url, Stream source)

I can't find a Stream implementation in the .NET framework that offers this capability.

This is not a case for Stream.CopyTo as both APIs implement their own read and write IO loops. If either of the APIs had returned a custom Stream implementation then Stream.CopyTo would be the answer, but since they both accept an input Stream for reading or writing, one has to implement a custom Stream that acts as both a sink (for the downloader) and a source (for the uploader) and puts back-pressure on the slower of the two to keep memory buffers from growing out of control.

James Dunne
  • 3,607
  • 3
  • 24
  • 29
  • 1
    Possible duplicate of [How do I copy the contents of one stream to another?](https://stackoverflow.com/questions/230128/how-do-i-copy-the-contents-of-one-stream-to-another) – cramopy Oct 06 '17 at 16:41
  • Not so because as the user of the APIs, I do not have control over either API's read or write loop in this situation. I need a single `Stream` that acts as both a sink (for the download) and a source (for the upload). Writes should be buffered up for subsequent Reads. – James Dunne Oct 06 '17 at 16:55
  • @JamesDunne, How large are the contents of each stream and how fast do you expect these requests to come in? – dacke.geo Oct 06 '17 at 17:18
  • The timing is totally dependent on network performance. The files being transferred are very large, several gigabytes. It is not desirable to store them on the file system due to the extra time overhead that would incur. – James Dunne Oct 06 '17 at 17:40
  • The stream class is Net is different than an Ultrix stream. Net only has one location pointer so you can only write or read at one time. An Ultrix stream has two location pointers so you can write and read at same time. – jdweng Oct 06 '17 at 17:52
  • I have an implementation of my own using `AutoResetEvent`s to block and unblock the writer while the reader finishes up reading the last written buffer. Debugging is slow-going. I was wondering if anyone else had any solutions to this problem that could be easily shared. – James Dunne Oct 06 '17 at 17:55

0 Answers0