Creating a method that uses WebBrowser and returns HTML?

Question

How do you create a method that does this:
Login to a website, then read a (member-only) page and return the HTML.

I came up with this (which obviously doesn't work because I don't know how to make it return the page content)

public string LoginAndReadPage() {
    WebBrowser wb = new WebBrowser();
    wb.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(wb_DocumentCompleted);
    wb.Navigate("hxxp://mysite.com/login");
}

private async void wb_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
    if (wb.Url.ToString().Contains("login"))
    {
        wb.Document.GetElementsByTagName("input").GetElementsByName("email")[0].SetAttribute("value", _login);
        wb.Document.GetElementsByTagName("input").GetElementsByName("password")[0].SetAttribute("value", _password);
        wb.Document.GetElementsByTagName("button")[0].InvokeMember("click");
    }
    else if (wb.Url.ToString().Contains("dashboard"))
    {
        return wb.DocumentText; // I want to return the content of mysite.com/dashboard
    }
    else
    {
        await Task.Delay(1000); //wait for 1 second just to let the WB catch up
        wb.Navigate("hxxp://mysite.com/dashboard");
    }
}

Thanks in advance

@Edmad: I want to return it, probably from LoginAndReadPage method — Aximili, Apr 02 '17 at 06:18
@Niklas No, do you have another solution? (The web page uses AngularJS to load the content and OAuth) — Aximili, Apr 02 '17 at 07:18
Im not familiar with that, though check and see if this works? http://stackoverflow.com/questions/11118712/webclient-accessing-page-with-credentials — Niklas, Apr 02 '17 at 07:26
Thanks Niklas, it is definitely related but looks a lot more complicated than using a WebBrowser (although I never use WebBrowser either) and I don't think WebClient handles AJAX-loaded content — Aximili, Apr 02 '17 at 08:40

score 1 · Answer 1 · answered Apr 02 '17 at 09:36

What you're trying to do is called "scraping" or sometimes "webscraping". It's a big topic so I'd recommend googling it.

Potentially you could use something like Selenium via the C# Driver to do this as well. Selenium was designed for automated UI testing but it definitely has all the tools you need to do what you want.

Creating a method that uses WebBrowser and returns HTML?

1 Answers1