How do I increase the performance of HttpWebResponse?

Question

I am trying to build an application that sends and receives responses from a website.

None of the solutions I've read on Stack Overflow have solved my problem, so I think that my code could use optimization.

I have the following thread:

void DomainThreadNamecheapStart()
{
    while (stop == false)
    {
        foreach (string FromDomainList in DomainList.Lines)
        {
            if (FromDomainList.Length > 1)
            {
                // I removed my api parameters from the string
                string namecheapapi = "https://api.namecheap.com/foo" + FromDomainList + "bar";

                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(namecheapapi);
                request.Proxy = null;
                request.ServicePoint.Expect100Continue = false;

                HttpWebResponse response = (HttpWebResponse)request.GetResponse();
                StreamReader sr = new StreamReader(response.GetResponseStream());

                status.Text = FromDomainList + "\n" + sr.ReadToEnd();

                sr.Close();
            }
        }
    }
}

This thread is called when a button is clicked:

private void button2_Click(object sender, EventArgs e)
{
    stop = false;
    Thread DomainThread = new Thread(new ThreadStart(DomainThreadNamecheapStart));
    DomainThread.Start();
}

I only receive around 12 responses in 10 seconds using the above code. When I try to make the same request in JavaScript or using a simple IFrame, it's more than twice as fast. The Browser doesn't use multiple threads for the connection, it waits until one is finished and then starts the new one.

I tried setting request.Proxy = null;, but it had negligible impact.

I have noticed that HTTPS is 2-3 times slower than HTTP. Unfortunately, I have to use HTTPS. Is there anything I can do to make it faster?

- "C# is slower than browser!" - "Browser is written in C++!" — , Dec 28 '12 at 15:39
Side note: `status.Text = ...` looks suspiciously like you're accessing a UI component from a non-UI thread. — Damien_The_Unbeliever, Dec 28 '12 at 15:46
@H2CO3, I don't think that alone would cause it to be more than twice as slow. It's likely it's not the request but simply the way the code is written here. In fact, the OP isn't timing the request, unfortunately the OP is timing the overall process, and though that can lead you to the root cause, you still have to dive deeper. I think the real question is that is it **fast enough**. — Mike Perrenoud, Dec 28 '12 at 15:48
@MichaelPerrenoud Of course, and Java isn't slow anymore thanks to JIT, I was making a joke :) — , Dec 28 '12 at 15:49
I have verified the "https" is causing the problem. I tried this on another website that works with both "https" and without, "https" version is slower than "http", but in my case i can't replace https with http. — Tumay, Dec 28 '12 at 15:50
@Tumay, that's something you're going to have to live with because there will always be overhead with `https` due to encryption. — Mike Perrenoud, Dec 28 '12 at 15:51
The browser should suffer the https-slowdown as well, though. — Wutz, Dec 28 '12 at 15:55
@Wutz: I've not delved into it, but its possible that .NET and "the browser" use different https mechanisms.. this even becomes likely if "the browser" is not Internet Explorer. — Sam Axe, Dec 28 '12 at 16:01
Yeah i was using Chrome for the browser, sorry for not being clear. — Tumay, Dec 28 '12 at 16:03
@Wutz: My bet would be that the browser is reusing its connection pool - thus, skipping the initial HTTPS handshake. — OnoSendai, Dec 28 '12 at 16:04
@OnoSendai After reading your answer, you are probably right. There would still be a little overhead though, the en-/decryption of the message itself. — Wutz, Dec 28 '12 at 16:06
@Wutz - True, since the dec/cod routines would be added to the the beginning and end of the pipeline. — OnoSendai, Dec 28 '12 at 16:11

score 5 · Accepted Answer · answered Dec 28 '12 at 15:57

My bet would be on the aspect you pointed out - the HTTPS protocol.

The iteration between Client(browser) and Server for pure HTTP protocol is quite straightforward: Ask for the info, get info. If 1.0, close connection; if 1.1, it may stay alive for reuse. (Check image 1 for details.)

But when you do a HTTPS request, the initial protocol overhead is considerable (image 2); but, once the initial negotiation is done, some symmetric encryption takes place, and no further certificate negotiation is necessary, thus speeding up data transfer.

What I think the problem is, if you destroy the HTTPWebRequest object and creates a new one, the full HTTPS exchange takes place once again, slowing your iteration. (HTTPS + HTTP 1.1 Keepalive should do fine, though.)

So, suggestions: Switch to HTTP only, or reuse the connection objects.

And i hope it works for you. =)

(1) HTTP protocol handshake and response

(2) Initial HTTPS protocol handshake enter image description here

Thanks for the great explaination, it helped me to understand things better. It's not possible for me to remove https though, I will have to find a way to keep it alive and make it use the same connection objects. I am still new to c# so it will take me some time to find out how to do it, but i love the challenge! :) — Tumay, Dec 28 '12 at 16:03
You're surely welcome! There's a nice post here about HTTP vs. HTTPS performance, which should do for some light reading. Here it goes: http://stackoverflow.com/questions/149274/http-vs-https-performance — OnoSendai, Dec 28 '12 at 16:13

bwdeng · Answer 2 · 2012-12-28T15:57:16.140

2

Try to modify the System.Net.ServicePointManager.DefaultConnectionLimit value (the default value is 2).

Other reference (Performance Issues Part).

edited Dec 28 '12 at 15:57

answered Dec 28 '12 at 15:49

bwdeng

358
2
11

score 2 · Answer 3 · answered Dec 28 '12 at 17:51

try these, it helped me to improve the performance,

        ServicePointManager.Expect100Continue = false;
        ServicePointManager.DefaultConnectionLimit = 200;
        ServicePointManager.MaxServicePointIdleTime = 2000;
        ServicePointManager.SetTcpKeepAlive(false, 0, 0);

How do I increase the performance of HttpWebResponse?

3 Answers3