C# - upload file by chunks - bad last chunk size

Question

I am trying to upload large files to 3rd part service by chunks. But I have problem with last chunk. Last chunk would be always smaller then 5mb, but all chunks incl. the last have all the same size - 5mb My code:

int chunkSize = 1024 * 1024 * 5;
using (Stream streamx = new FileStream(file.Path, FileMode.Open, FileAccess.Read))
 {
    byte[] buffer = new byte[chunkSize];

    int bytesRead = 0;
    long bytesToRead = streamx.Length;

    while (bytesToRead > 0)
    {

        int n = streamx.Read(buffer, 0, chunkSize);

        if (n == 0) break;

        // do work on buffer...
        // uploading chunk ....
        var partRequest = HttpHelpers.InvokeHttpRequestStream
            (
                new Uri(endpointUri + "?partNumber=" + i + "&uploadId=" + UploadId),
                "PUT",
                 partHeaders,
                 buffer
            );  // upload buffer


        bytesRead += n;
        bytesToRead -= n;

    }
    streamx.Dispose();
 }

buffer is uploaded on 3rd party service.

The short answer is that you should upload `n` bytes, not `chunkSize` bytes. — mjwills, Jul 19 '18 at 12:21
Unclear where `chunkSize` comes out from, but it should be `streamx.Read(buffer, 0, buffer.Length);` — xanatos, Jul 19 '18 at 12:21
The bug is in `// do work on buffer...` or `// uploading chunk ....` - you need to show us that code. — mjwills, Jul 19 '18 at 12:24

Karel_cz · Answer 1 · 2018-07-19T13:01:00.967

Solved, someone posted updated code in comment, but after some seconds deleted this comment. But there was solution. I added this part after

if (n == 0)

this code, which resizes last chunk on the right size

// Let's resize the last incomplete buffer
if (n != buffer.Length)
    Array.Resize(ref buffer, n);

Thank you all.

I post full working code:

int chunkSize = 1024 * 1024 * 5;
using (Stream streamx = new FileStream(file.Path, FileMode.Open, FileAccess.Read))
 {
    byte[] buffer = new byte[chunkSize];

    int bytesRead = 0;
    long bytesToRead = streamx.Length;

    while (bytesToRead > 0)
    {

        int n = streamx.Read(buffer, 0, chunkSize);

        if (n == 0) break;

        // Let's resize the last incomplete buffer
        if (n != buffer.Length)
           Array.Resize(ref buffer, n);

        // do work on buffer...
        // uploading chunk ....
        var partRequest = HttpHelpers.InvokeHttpRequestStream
            (
                new Uri(endpointUri + "?partNumber=" + i + "&uploadId=" + UploadId),
                "PUT",
                 partHeaders,
                 buffer
            );  // upload buffer


        bytesRead += n;
        bytesToRead -= n;

    }

 }

This is not a good idea. As per @xanatos' earlier comment - `A Stream could return 1 byte at a time... after this you would have a small buffer forever`. — mjwills, Jul 19 '18 at 12:57
What you **should** do is upload only the first `n` bytes of `buffer`. `var g = buffer; if (n < buffer.Length) g = g.Take(n).ToArray();` Then upload `g` inside of `buffer`. — mjwills, Jul 19 '18 at 12:58
@mjwills ok and how would you solve it? Upload n bytes from buffer? — Karel_cz, Jul 19 '18 at 12:59

C# - upload file by chunks - bad last chunk size

1 Answers1

Linked