Get web page using HtmlAgilityPack.NETCore

Question

I used the HtmlAgilityPack for work with html pages. Previously I did this:

HtmlWeb web = new HtmlWeb();
HtmlDocument document = web.Load(url);
var nodes = document.DocumentNode.SelectNodes("necessary node");

but now i need to use the HtmlAgilityPack.NETCore where HtmlWeb is absent. What should i use instead HtmlWeb to have the same result?

score 8 · Accepted Answer · edited Sep 07 '17 at 19:46

Use the HttpClient as a new way to interact with remote resources via http.

As for your solution, you probably need to use the async methods here for non-blocking your thread, instead of .Result usage. Also note that HttpClient was meant to be used from different threads starting from .Net 4.5, so you should not recreate it each time:

// instance or static variable
HttpClient client = new HttpClient();

// get answer in non-blocking way
using (var response = await client.GetAsync(url))
{
    using (var content = response.Content)
    {
        // read answer in non-blocking way
        var result = await content.ReadAsStringAsync();
        var document = new HtmlDocument();
        document.LoadHtml(result);
        var nodes = document.DocumentNode.SelectNodes("Your nodes");
        //Some work with page....
    }
}

Great article about async/await: Async/Await - Best Practices in Asynchronous Programming by @StephenCleary | March 2013

score 4 · Answer 2 · edited Apr 30 '19 at 11:54

I had the same problem in Visual Studio code with netcoreapp1.0. Ended up using HtmlAgilityPack version 1.5.0-beta5 instead.

Remember to add:

using HtmlAgilityPack;
using System.Net.Http;
using System.IO;

I did it like this:

HttpClient hc = new HttpClient(); 
HttpResponseMessage result = await hc.GetAsync($"http://somewebsite.com"); 
Stream stream = await result.Content.ReadAsStreamAsync(); 
HtmlDocument doc = new HtmlDocument(); 
doc.Load(stream); 
HtmlNodeCollection nodes = doc.DocumentNode.SelectNodes("//div[@class='whateverclassyouarelookingfor']");

AlexCOM · Answer 3 · 2017-04-12T11:36:41.797

1

I wrote this and it's working. Is this a good way to solve my problem?

using (HttpClient client = new HttpClient())
{
    using (HttpResponseMessage response = client.GetAsync(url).Result)
    {
        using (HttpContent content = response.Content)
        {
            string result = content.ReadAsStringAsync().Result;
            HtmlDocument document = new HtmlDocument();
            document.LoadHtml(result);
            var nodes = document.DocumentNode.SelectNodes("Your nodes");
            //Some work with page....
        }
    }
}

edited Apr 12 '17 at 11:36

answered Apr 12 '17 at 11:24

AlexCOM

129
2
2
11

It's interesting for me. Why we cannot use await for : client.GetAsync(url) – alerya Dec 14 '17 at 06:32

score -1 · Answer 4 · answered Apr 12 '17 at 09:00

-1

You can use HttpClient to get the content of the page.

answered Apr 12 '17 at 09:00

Ivan Ičin

9,672
5
36
57

I'm looking for it but can't find. Can you show me how use it? – AlexCOM Apr 12 '17 at 10:13
Did you mean something like this? `WebClient client = new WebClient(); client.DownloadFile(url, path);` i would like to work with file without saving. – AlexCOM Apr 12 '17 at 10:18

Get web page using HtmlAgilityPack.NETCore

4 Answers4

Linked