.NET: WebBrowser, WebClient, WebRequest, HTTPWebRequest... ARGH!

Question

In the System.Net namespace, there are very many different classes with similar names, such as:

• WebBrowser and WebClient
• WebRequest and HTTPWebRequest
• WebResponse and HTTPWebResponse

Those are the main ones I'm curious about.

What is each one's function? How are they different from one another?

Also, in what cases would you use which?

Answer 1

WebBrowser is actually in the System.Windows.Forms namespace and is a visual control that you can add to a form. It is primarily a wrapper around the Internet Explorer browser (MSHTML). It allows you to easily display and interact programmatically with a web page. You call the Navigate method passing a web URL, wait for it to complete downloading and display and then interact with the page using the object model it provides.

HttpWebRequest is a concrete class that allows you to request in code any sort of file over HTTP. You usually receive it as a stream of bytes. What you do with it after that is up to your application.

HttpWebResponse allows you to process the response from a web server that was previously requested using HttpWebRequest.

WebRequest and WebResponse are the abstract base classes that the HttpWebRequest and HttpWebResponse inherit from. You can't create these directly. Other classes that inherit from these include Ftp and File classes.

WebClient I have always seen as a nice helper class that provides simpler ways to, for example, download or upload a file from a web url. (eg DownloadFile and DownloadString methods). I have heard that it actually uses HttpWebRequest / HttpWebResponse behind the scenes for certain methods.

If you need more fine grained control over web requests and responses, HttpWebRequest / HttpWebResponse are probably the way to go. Otherwise WebClient is generally simpler and will do the job.

Answer 2

I don't know of any System.Net.WebBrowser, but WebClient is basically a class that lets you easily download files (including html pages) from the web into memory or even directly to file. A basic code sample looks like this:

string html;
using (var wc = new WebClient())
{
    html = wc.DownloadString("http://stackoverflow.com/questions/1780679/");
}

You can do a lot with WebClient, but there are some limitations. If you need to do some serious web scraping, you'll need to get lower level. That's where the HttpWebRequest/HttpWebResponse come in. You can use them to send any request a normal web browser might send, in any sequence. For example, you may need to authenticate with a web site before you can request the page you really want, and WebClient might not be able to do that. HttpWebRequest will.

Now, there is one other option. System.Windows.Forms.WebBrowser is a control designed to place on a form. It basically wraps the engine used in Internet Explorer to provide all the capabilities of a web browser. You need to be careful using this for general scraping: it's not portable (bad for mono), uses a lot of resources, has similar security issues as running a full browser, and has side-effects such as potentially leaking popup windows. The control is best used in a form to connect to a specific known web resource. For example, you may have a Windows Forms app for sale, and web app where you sell it for download. You might provide a WebBrowser control that shows a few pages on this web site specifically intended for view in your app that allows users to purchase in-app upgrades.

Answer 3

WebRequest and WebResponse are abstract classes. HTTPWebRequest and HTTPWebResponse are implementations of them.

Answer 4

WebClient is quite a neat way to fetch the HTML page. Here is the code snippet to download a response string.

   string getHtmlPageUsingWC(string strQuery, System.Net.WebProxy proxy = null)
    {
        string strResponse = String.Empty;
        using (WebClient wc = new WebClient())
        {
            wc.Encoding = Encoding.UTF8;
            IWebProxy wp = WebRequest.DefaultWebProxy;
            wp.Credentials = CredentialCache.DefaultCredentials;
            wc.Proxy = wp;
            wc.Headers.Add("Accept-Language:en");

            NameValueCollection nvc = new NameValueCollection();
            nvc.Add("q", strQuery);

            wc.QueryString.Add(nvc);

            try
            {
                strResponse = wc.DownloadString(m_strURL);
            }
            catch (Exception ex)
            {
                strResponse = "Request Declined: " + ex.Message;
                Console.WriteLine(ex.Message);
            }
        }

        return strResponse;
    }