Downloading file with webscraper sometimes yields HTML page

Question

I have a C# application to access data from a third party website. I'm using WebClient and I call DownloadData and process the bytes(DownloadFile would yield similar results).

This worked fine until recently. Now, more often than not, instead of the desired XML file, I get an html page similar to:

<!-- _localBinding -->
<!-- _lcid="1033" _version="" -->
<html>
    <head>
        <meta HTTP-EQUIV="Content-Type" content="text/html; charset=utf-8" />
        <meta HTTP-EQUIV="Expires" content="0" />
        <noscript>
            <meta http-equiv="refresh" content="0; url=/_layouts/spsredirect.aspx?noscript=1" />
        </noscript>
        <script language="javascript" src="/_layouts/1033/init.js"></script>
        <script language="javascript" src="/_layouts/1033/core.js"></script>
        <script language="javascript">

        var requestedUrl = escapeProperly(window.location.href);
        STSNavigate("/CustomErrors/ErrorPage.aspx?oldUrl=" + requestedUrl);

    </script>
    </head>
    <body></body>
</html>

I'm stumped as to why this went from consistently working to consistently giving me problems. Sometimes, I can still get the expected XML file. Do you have an idea of what this issue might be and how I can fix it?

It might be worth noting that if I were to click the download link manually in a browser, a javascript window would pop up and close before the file downloads. I'm using WebBrowser to intercept the non-static URL of that popup and pass that link (and headers/cookie information) to WebClient. The link would look something like https://foo.bar/Something.axd?Session=1234vv1234Format=XML

Thanks for your time

score 0 · Accepted Answer · answered Oct 23 '14 at 20:44

0

It looks like the third-party page is throwing an exception. This could be due to parameters you are passing it, or it could be a problem on their end. The best course of action would be to contact the provider and see what errors are being logged (hopefully they are logging them somewhere). Otherwise you are just going to be guessing at what you are doing wrong.

Is it possible that there are encoded\escaped characters in the session querystring param that are not being handled properly? That would be my first guess since there aren't any real error details.

answered Oct 23 '14 at 20:44

DVK

2,726
1
17
20

Thank you for the quick response! It's reassuring. Their site seems to be pretty buggy but this is also my first time doing something like this. As for the querystring, the only special characters I see in the URL are = + and & – reborn programmer Oct 23 '14 at 21:05
I used to work with payment gateways a lot and we'd run into oddities like this. I found early on you could spend three days guessing at a problem like this and then end up resolving it with a single email to their support team. Trying to debug an error without being able to see the error is very difficult. – DVK Oct 23 '14 at 21:50

score 0 · Answer 2 · answered Oct 23 '14 at 21:10

0

If it's throwing an error, I'd attempt to collect the HTTP status code, you can do this by using the HttpWebRequest and HttpWebResponse (I believe these pre-dated WebClient). There's tricky ways to still use WebClient and retrieve the status code as well, I'd check the content type as well before you actually download.

In any condition, even if serving an error page, the server should respond with an appropriate HTTP status code. (At least in a perfect world).

answered Oct 23 '14 at 21:10

jamesbar2

614
1
9
20

I used Dmitry's code http://stackoverflow.com/questions/3574659/how-to-get-status-code-from-webclient to determine status code. Before I download: 0. After: 200 OK. Downloaded data: the HTML page up above. – reborn programmer Oct 23 '14 at 21:33
A lot of times when a custom errors page is used the status code is lost. Even ASP.NET CustomErrors will mask the status code. – DVK Oct 23 '14 at 21:49

Downloading file with webscraper sometimes yields HTML page

2 Answers2