Save HTML Source Code to string in WinForms Application

Question

I need to get the source code from a website, which is structured in frames.

I already have a Windows Forms Application which has the WebBrowser function integrated into it.
When I do a right click and select "View Source" it opens a new text document with exactly the info I need.

I already tried webBrowser.Document, webBrowser.DocumentText and webBrowser.DocumentStream, but all of these only give me other info, I don't need.

The website is not static (it's a chat) and it does not do sessions, thus I can't use Webclient.DownloadFile.
I need to have an ongoing connection to the website for several hours without refreshing the website. I don't see a way around using the webBrowser in Windows Forms.

As requested, this is the website, I am talking about: http://server2.webkicks.de/stackoverflow-test/
You can just log in as a guest, by filling some username in the third textbox.

Answer 1

As you wish to get the dynamic html content, and webBrowser.Document, webBrowser.DocumentText and webBrowser.DocumentStream are not working to your wish.

Here's the trick: You can always run your custom JavaScript code from C#. And here's how you can get the current HTML in your WebBrowser control:

webBrowser.Document.InvokeScript("eval", new string[]{"document.body.outerHTML"});

Refer to How to inject Javascript in WebBrowser control?.

Update

For iframe inside your document, you can try the following:

webBrowser.Document.InvokeScript("eval", new string[]{"document.querySelector(\"iframe\").contentWindow.document.documentElement.outerHTML"});

Another update

As your site contains the frame instead of iframe, here is how you can get the html content of that frame:

webBrowser.Document.InvokeScript("eval", new string[]{"document.querySelector(\"frame[name='mainframe'\").contentWindow.document.documentElement.outerHTML"});

Final tested and working update

querySelector is not working in WebControl. So the workaround is: Provide some id to your <frame>, and fetch that <frame> element using that id. Here is how you can achieve your task.

HtmlElement frame = webBrowser1.Document.GetElementsByTagName("frame").Cast<HtmlElement>().FirstOrDefault(m => m.GetAttribute("name") == "mainframe");
if (frame != null)
{
    frame.Id = "RandID_" + DateTime.Now.Ticks;
    string html = webBrowser1.Document.InvokeScript("eval", new string[] { "document.getElementById('" + frame.Id + "').contentWindow.document.documentElement.outerHTML" }).ToString();
    Console.WriteLine(html);
}
else
{
    MessageBox.Show("Frame not found");
}

Answer 2

If your website target use the ssl protocol (https) you can try adding the user-agent like this :

using (WebClient myWebClient = new WebClient())
                            {
                                myWebClient.Headers.Add("User-Agent: Other");               
                                myWebClient.DownloadFile(new System.Uri("https://mywebsite.com//somefile"), "D:\\temp\\somefile");
                            }

If your website target needs login, then you log in into your websitetarget in chrome and use EditThisCookie extension to copy your cookies and try this one :

using (WebClient myWebClient = new WebClient())
                            {
                                myWebClient.Headers.Add("User-Agent: Other");
                                myWebClient.Headers.Add(HttpRequestHeader.Cookie, "mycookies copies from EditThisCookie");
                                myWebClient.DownloadFile(new System.Uri("https://mywebsite.com//somefile"), "D:\\temp\\somefile");
                            }