HtmlAgilityPack obtain Title and meta

Question

I try to practice "HtmlAgilityPack ", but I am having some issues regarding this. here's what I coded, but I can not get correctly the title and the description of a web page ... If someone can enlighten me on my mistake :)

...
public static void Main(string[] args)
    {
        string link = null;
        string str;
        string answer;

        int curloc; // holds current location in response 
        string url = "http://stackoverflow.com/";

        try
        {

            do
            {
                HttpWebRequest HttpWReq = (HttpWebRequest)WebRequest.Create(url);
                HttpWReq.UserAgent = @"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5";
                HttpWebResponse HttpWResp = (HttpWebResponse)HttpWReq.GetResponse();
                //url = null; // disallow further use of this URI 
                Stream istrm = HttpWResp.GetResponseStream();
                // Wrap the input stream in a StreamReader. 
                StreamReader rdr = new StreamReader(istrm);

                // Read in the entire page. 
                str = rdr.ReadToEnd();

                curloc = 0;
                //WebPage result;
                do
                {
                    // Find the next URI to link to. 
                    link = FindLink(str, ref curloc); //return the good link
                    Console.WriteLine("Title found: " + curloc);
                    //title = Title(str, ref curloc);

                    if (link != null)
                    {
                        Console.WriteLine("Link found: " + link);
                        using (System.Net.WebClient client = new System.Net.WebClient())
                        {
                            HtmlDocument htmlDoc = new HtmlDocument();
                            var html = client.DownloadString(url);
                            htmlDoc.LoadHtml(link); //chargement de HTMLAgilityPack
                            var htmlElement = htmlDoc.DocumentNode.Element("html");

                            HtmlNode node = htmlDoc.DocumentNode.SelectSingleNode("//meta[@name='description']");
                            if (node != null)
                            {
                                string desc = node.GetAttributeValue("content", "");
                                Console.Write("DESCRIPTION: " + desc);
                            }
                            else
                            {
                                Console.WriteLine("No description");
                            }

                            var titleElement =
                                                htmlDoc.DocumentNode
                                                   .Element("html")
                                                   .Element("head")
                                                   .Element("title");
                            if (titleElement != null)
                            {
                                string title = titleElement.InnerText;
                                Console.WriteLine("Titre: {0}", title);
                            }
                            else
                            {
                                Console.WriteLine("no Title");
                            }
                            Console.Write("Done");
                        }
                        Console.Write("Link, More, Quit?");
                        answer = Console.ReadLine();
                    }
                    else
                    {
                        Console.WriteLine("No link found.");
                        break;
                    }
                } while (link.Length > 0);

                // Close the Response.
                HttpWResp.Close();
            } while (url != null); 
        }
catch{ ...}

Thanks in advance :)

There is meta tag on stackoverflow? My browser doesn't shows it in source code of page — Viktor Lova, Jul 12 '13 at 01:45
About title: just try htmlDoc.DocumentNode.SelectSingleNode("//title") — Viktor Lova, Jul 12 '13 at 01:47
yes, there are not meta tag on stackoverflow ... and with "HtmlNode node = htmlDoc.DocumentNode.SelectSingleNode("//title");" the result is empty ... — pi-2r, Jul 12 '13 at 02:02

prospector · Accepted Answer · 2016-12-15T22:43:59.773

20

Go about it this way:

HtmlNode mdnode = htmlDoc.DocumentNode.SelectSingleNode("//meta[@name='description']");

              if (mdnode != null)
              {
                 HtmlAttribute desc;

                 desc = mdnode.Attributes["content"];
                 string fulldescription = desc.Value;
                 Console.Write("DESCRIPTION: " + fulldescription);
              }

edited Dec 15 '16 at 22:43

answered Jul 12 '13 at 16:47

prospector

3,389
1
23
40

2

Should it not be "if (mdnode != null)" on line 3? – XzaR Dec 15 '16 at 10:24
@XzaR yep, it appears so – prospector Dec 15 '16 at 22:44

fcuesta · Answer 2 · 2013-07-12T02:48:53.070

1

I think your problem is here:

htmlDoc.LoadHtml(link); //chargement de HTMLAgilityPack

It should be:

 htmlDoc.LoadHtml(html); //chargement de HTMLAgilityPack

LoadHtml expects a string with the HTML source, not the url.

And probably you want to change:

var html = client.DownloadString(url);

to

var html = client.DownloadString(link);

edited Jul 12 '13 at 02:48

answered Jul 12 '13 at 02:41

fcuesta

4,429
1
18
13

score 1 · Answer 3 · answered Jul 12 '13 at 03:37

1

Have you used a breakpoint and gone line for line to see where the error might be occurring?

If you have, then Try something like this:

string result = string.Empty;
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.google.com");
        request.Method = "GET";
        try
        {
            using (var stream = request.GetResponse().GetResponseStream())

            using (var reader = new StreamReader(stream, Encoding.UTF8))
            {
                result = reader.ReadToEnd();
            }
        }
       HtmlAgilityPack.HtmlDocument htmlDoc = new HtmlAgilityPack.HtmlDocument();
       htmlDoc.LoadHtml(result);

Then carry over the rest of your code below the htmlDoc.LoadHtml

answered Jul 12 '13 at 03:37

prospector

3,389
1
23
40

yes, but I the same problem when I try to obtain the "Website_title". – pi-2r Jul 12 '13 at 14:34
string desc = website_title.InnerText; – prospector Jul 12 '13 at 15:41
thanks it's work fine for the Title, but I've not result for the description. I try this: string desc = description.GetAttributeValue("content=", ""); – pi-2r Jul 12 '13 at 16:35

score 1 · Answer 4 · answered Dec 06 '13 at 08:45

 [HttpPost]
    public ActionResult Create(WebSite website)
    {



        string desc = HtmlAgi(website.Url, "description");
        string keyword = HtmlAgi(website.Url, "Keywords");

        if (ModelState.IsValid)
        {

            var userId = ((CustomPrincipal)User).UserId;
            r.Create(new WebSite
            {
                Description = desc,
                Tags = keyword,
                Url = website.Url,
                UserId = userId,
                Category = website.Category

            });

            return RedirectToAction("Index");
        }

        return View(website);
    }

score 0 · Answer 5 · answered Dec 06 '13 at 08:41

    public string HtmlAgi(string url, string key)
    {
        //string.Format
        var Webget = new HtmlWeb();
        var doc = Webget.Load(url);
        HtmlNode ourNode = doc.DocumentNode.SelectSingleNode(string.Format("//meta[@name='{0}']", key));
        if (ourNode != null)
        {
            return ourNode.GetAttributeValue("content", "");
        }
        else
        {
            return "not fount";
        }

    }

HtmlAgilityPack obtain Title and meta

5 Answers5

Linked