Parallel Blob Upload throwing 404 Bad Request intermittently

Question

I have a very simple service,

public class AzureService : IAzureService
{
    private readonly CloudBlobContainer _container;
    public AzureService(ISettings settings)
    {
        var storageAccount = CloudStorageAccount.Parse(settings.BlobConnectionString);            
        var blobClient = storageAccount.CreateCloudBlobClient();
        _container = blobClient.GetContainerReference(settings.BlobContainerName);
    }

    public Task UploadBlobAsync(string fileName, Stream stream)
    {
        var blob = _container.GetBlockBlobReference(fileName);
        return blob.UploadFromStreamAsync(stream);
    }

    public Task DeleteBlobAsync(string fileName)
    {
        var blob = _container.GetBlockBlobReference(fileName);
        return blob.DeleteAsync();
    }
}

This method is called from,

    public Task SaveAllAsync(Dictionary<string, Stream> images)
    {
        var tasks = new List<Task>();
        foreach (var image in images)
        {
            var fileName = image.Key;
            var stream = image.Value;
            var task = _azureService.UploadBlobAsync(fileName, stream);
            tasks.Add(task);
        }
        return Task.WhenAll(tasks);
    }

My stream is HttpPostedFileBase.InputStream. Sometimes it works and sometimes I get The remote server returned an error: (400) Bad Request.. If I put a break-point it works as well.

400 is definitely a server error saying the client sent a bad request. Try using Fiddler to see what's different between the good and bad requests. — Paulo Morgado, Jun 15 '14 at 18:59
@PauloMorgado, If I `await UploadFromStreamAsync` then it will work — Imran Qadir Baksh - Baloch, Jun 15 '14 at 19:45
Awaiting `UploadFromStreamAsync` shouldn't have any effect. Have you checked the HTTP content with Fiddler? — Paulo Morgado, Jun 16 '14 at 09:24
There isn't enough information. When does the bad request occur? Certain times? Randomly? After a specific amount of time? Do you catch an exception? If so any inner messages you see. Have you looked via fiddler to so you requests are always valid? Can your server take multiple uploads at one? — Yuval Itzchakov, Jun 18 '14 at 05:26
@YuvalItzchakov, now it's happening most of time. I am calling SaveAllAsync from controller like await `service.SaveAllAsync` — Imran Qadir Baksh - Baloch, Jun 20 '14 at 11:22
@user960567 Have you tried looking at the requests sent to your server via Fiddler? — Yuval Itzchakov, Jun 22 '14 at 15:03

Zakos · Accepted Answer · 2014-11-11T14:52:21.787

I had the same problem , I tried to upload 20 + images in 1 strike , single threaded works , multi threaded using await Task.WhenAll failed with "The remote server returned an error: (400) Bad Request."

see RequestInformation inside Microsoft.WindowsAzure.Storage.StorageException which is thrown from Upload[xxx]Async methods for more detailed information.
At first , RequestInformation said something about a MD5 problem with error code of "Md5Mismatch" , buy my intuition said otherwise because single thread works like a charm , and then .. I found it... DefaultRequestOptions.ParallelOperationThreadCount on CloudBlobClient object and problem sovled.
BlobRequestOptions Members MSDN

    private CloudBlobContainer ConnectToImageContainer()
    {
        var credentials = new StorageCredentials(AccountName, ImagesContainerKey);
        var account = new CloudStorageAccount(credentials, useHttps: true);
        var client = account.CreateCloudBlobClient();
        client.DefaultRequestOptions.ParallelOperationThreadCount = 64; // max value
        client.DefaultRequestOptions.SingleBlobUploadThresholdInBytes = 67108864; // max value
        var container = client.GetContainerReference(ImagesContainerName);
        return container;
    }

score 0 · Answer 2 · edited May 23 '17 at 11:57

The behaviour you are describing sounds very much like a threading problem (i.e. if you breakpoint the code it works fine as it's effectively single-threaded at that time) resulting in incomplete or invalid data being sent to the Azure Storage API.

Your "var image" definition may behave unexpectedly in a multi-threaded environment (If you utilised ReSharper it will highlight this variable and advise to change the code because it is potentially unsafe).

Have a read of this SO post to understand a little more and how you might better implement your code.

The foreach identifier and closures

Parallel Blob Upload throwing 404 Bad Request intermittently

2 Answers2

Linked