HttpWebRequest.GetResponse() keeps getting timed out

Question

i wrote a simple C# function to retrieve trade history from MtGox with following API call:

https://data.mtgox.com/api/1/BTCUSD/trades?since=<trade_id>

documented here: https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Multi_currency_trades

here's the function:

string GetTradesOnline(Int64 tid)
{
    Thread.Sleep(30000);

    // communicate
    string url = "https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString();
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
    StreamReader reader = new StreamReader(response.GetResponseStream());

    string json = reader.ReadToEnd();
    reader.Close();
    reader.Dispose();
    response.Close();

    return json;
}

i'm starting at tid=0 (trade id) to get the data (from the very beginning). for each request, i receive a response containing 1000 trade details. i always send the trade id from the previous response for the next request. it works fine for exactly 4 requests & responses. but after that, the following line throws a "System.Net.WebException", saying that "The operation has timed out":

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

here are the facts:

• catching the exception and retying keeps causing the same exception
• the default HttpWebRequest .Timeout and .ReadWriteTimeout are already high enough (over a minute)
• changing HttpWebRequest.KeepAlive to false didn't solve anything either
• it seems to always work in the browser even while the function is failing
• it has no problems retrieveing the response from https://www.google.com
• the amount of successful responses before the exceptions varies from day to day (but browser always works)
• starting at the trade id that failed last time causes the exception immediately
• calling this function from the main thread instead still caused the exception
• running it on a different machine didn't work
• running it from a different IP didn't work
• increasing Thread.Sleep inbetween requests does not help

any ideas of what could be wrong?

Answer 1

I had the very same issue. For me the fix was as simple as wrapping the HttpWebResponse code in using block.

using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
    // Do your processings here....
}

Details: This issue usually happens when several requests are made to the same host, and WebResponse is not disposed properly. That is where using block will properly dispose the WebResponse object properly and thus solving the issue.

Answer 2

There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:

request.Timeout = Timeout.Infinite;
request.KeepAlive = true;

Try something like this...

Answer 3

I just had similar troubles calling a REST Service on a LINUX Server thru ssl. After trying many different configuration scenarios I found out that I had to send a UserAgent in the http head.

Here is my final method for calling the REST API.

        private static string RunWebRequest(string url, string json)
    {
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);

        // Header
        request.ContentType = "application/json";
        request.Method = "POST";
        request.AllowAutoRedirect = false;
        request.KeepAlive = false;
        request.Timeout = 30000;
        request.ReadWriteTimeout = 30000;
        request.UserAgent = "test.net";
        request.Accept = "application/json";
        request.ProtocolVersion = HttpVersion.Version11;
        request.Headers.Add("Accept-Language","de_DE");
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
        ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
        byte[] bytes = Encoding.UTF8.GetBytes(json);
        request.ContentLength = bytes.Length;
        using (var writer = request.GetRequestStream())
        {
            writer.Write(bytes, 0, bytes.Length);
            writer.Flush();
            writer.Close();
        }

        var httpResponse = (HttpWebResponse)request.GetResponse();
        using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
        {
            var jsonReturn = streamReader.ReadToEnd();
            return jsonReturn;
        }
    }

Answer 4

This is not a solution, but just an alternative: These days i almost only use WebClient instead of HttpWebRequest. Especially WebClient.UploadString for POST and PUT and WebClient.DownloadString. These simply take and return strings. This way i don't have to deal with streams objects, except when i get a WebException. i can also set the content type with WebClient.Headers["Content-type"] if necessary. The using statement also makes life easier by calling Dispose for me.

Rarely for performance, i set System.Net.ServicePointManager.DefaultConnectionLimit high and instead use HttpClient with it's Async methods for simultaneous calls.

This is how i would do it now

string GetTradesOnline(Int64 tid)
{
    using (var wc = new WebClient())
    {
        return wc.DownloadString("https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString());
    }
}

2 more POST examples

// POST
string SubmitData(string data)
{
    string response;
    using (var wc = new WebClient())
    {
        wc.Headers["Content-type"] = "text/plain";
        response = wc.UploadString("https://data.mtgox.com/api/1/BTCUSD/trades", "POST", data);
    }

    return response;
}

// POST: easily url encode multiple parameters
string SubmitForm(string project, string subject, string sender, string message)
{
    // url encoded query
    NameValueCollection query = HttpUtility.ParseQueryString(string.Empty);
    query.Add("project", project);
    query.Add("subject", subject);

    // url encoded data
    NameValueCollection data = HttpUtility.ParseQueryString(string.Empty);
    data.Add("sender", sender);
    data.Add("message", message);

    string response;
    using (var wc = new WebClient())
    {
        wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
        response = wc.UploadString( "https://data.mtgox.com/api/1/BTCUSD/trades?"+query.ToString()
                                  , WebRequestMethods.Http.Post
                                  , data.ToString()
                                  );
    }

    return response;
}

Error handling

try
{
    Console.WriteLine(GetTradesOnline(0));

    string data = File.ReadAllText(@"C:\mydata.txt");
    Console.WriteLine(SubmitData(data));

    Console.WriteLine(SubmitForm("The Big Project", "Progress", "John Smith", "almost done"));
}
catch (WebException ex)
{
    string msg;
    if (ex.Response != null)
    {
        // read response HTTP body
        using (var sr = new StreamReader(ex.Response.GetResponseStream())) msg = sr.ReadToEnd();
    }
    else
    {
        msg = ex.Message;
    }

    Log(msg);
}

Answer 5

For what it's worth, I was experiencing the same issues with timeouts every time I used it, even though calls went through to the server I was calling. The problem in my case was that I had Expect set to application/json, when that wasn't what the server was returning.