Download a file from a variable path every 15 minutes

Question

I'm working on a console application which is scheduled in windows scheduler to run every 15 minutes which when ran downloads a file from a public website using WebClient.

string Url1 = "http://www2.epa.gov/sites/production/files/" + DateTime.Now.Year + "-" + DateTime.Now.Month.ToString("d2")+ "/rindata.csv";
WebClient webClient = new WebClient();
webClient.DownloadFile(Url1, filename);

The above code works fine, but the above URL might or might not change every month randomly which cause my application throw 404 Exception.

Example

Consider the URL to be http://www2.epa.gov/sites/production/files/2015-09/rindata.csv and the variable part of the URL is 2015-09 which contains the data regarding September and it might change to 2015-10 for October if there any data change for that month but there no pattern of when or whether it changes everymonth.

May I know a better way to handle this?

If you know it is the date, you might consider computing the url dynamically. You can check for http response codes and act on that as you wish (check this post: http://stackoverflow.com/questions/3574659/how-to-get-status-code-from-webclient) — Rik, Nov 02 '15 at 15:37
How about surrounding this by a try/catch block and silently ignoring `WebException`s. (It does however look like doing this every 15 minutes is an overkill, since this looks like a static monthly report.) — steinar, Nov 02 '15 at 15:41
If you don't know the date currently being used, then you really have no other option than to make multiple requests and check the response codes you get and act accordingly. This might result in a high number of requests, so make sure to sleep for a few seconds between requests. — HaukurHaf, Nov 02 '15 at 15:41

score 0 · Answer 1 · 2015-11-02T15:19:37.873

0

To make it download every 15 minutes you can use a timer, set its interval to 15 minutes(in miliseconds), and put that code in the tick. Regarding the change of URL, I don´t realize of a better way to do it.

edited Nov 02 '15 at 15:19

answered Nov 02 '15 at 15:15

issue is not about making it download every 15 minutes but its with change of URL as I mentioned it in my question – DoIt Nov 02 '15 at 15:17
I have edited my question with an example. PLease check – DoIt Nov 02 '15 at 15:23
I would try using an if-statement. Example: If you get the 404 error, change it to the next month. Is the best way that I can realize of at the moment. – Nov 02 '15 at 15:28
Agreed. Your best bet would be to trap failures and retry, with some logic in there telling you what dates to use, etc. I'm guessing you don't want to accidentally go too far in the future in case you happen to have a good request fail for some reason... – Broots Waymb Nov 02 '15 at 15:44
@DangerZone, exactly it might not be the best way to do it out there. But for me is the most simple one, and when I code I look for the easiest way ;) Please, if that worked for you give me a positive vote. :) – Nov 02 '15 at 16:17

n1FF · Answer 2 · 2015-11-02T15:41:51.067

It sounds like the URL won't necessarily update every month, so if this is the case don't re-evaluate the string every 15 minutes. So on first run set

string Url1 = "http://www2.epa.gov/sites/production/files/" + DateTime.Now.Year + "-" + DateTime.Now.Month.ToString("d2")+ "/rindata.csv";

you'll have to save this working value somewhere like the config file, then you can keep reusing this value until it fails, so every 15 minutes only run

WebClient webClient = new WebClient();
webClient.DownloadFile(Url1, filename);

instead of evaluating the URL again. When it fails then re-evaluate it to the current month again.

if (failed)
{
    string Url1 = "http://www2.epa.gov/sites/production/files/" + DateTime.Now.Year + "-" + DateTime.Now.Month.ToString("d2")+ "/rindata.csv";
}

then overwrite the saved value.

More info on saving to a settings file: https://msdn.microsoft.com/en-us/library/aa730869(VS.80).aspx

score 0 · Answer 3 · answered Nov 02 '15 at 15:49

I regards to the Date change, I would probably do a backwards search from 12 to 1. Since the newest data is all you're interested in, a higher month that doesn't return a 404 would always be the freshest data. A simple loop that checks for a 404 would be fine if you have no other way to know what the URL is. This is a very basic example but the concept should be sound.

        for (int i = 12; i > 1; i--)
        {
            string folder = string.Format("{0}-{1}", DateTime.Now.Year + "-" + i.ToString().PadLeft(2, '0'));
            string Url1 = "http://www2.epa.gov/sites/production/files/" + folder + "/rindata.csv";
            try
            {
                using (WebClient client = new WebClient())
                {
                    client.DownloadFile(Url1, "/rindata.csv");
                }
            }
            catch (Exception e)
            {
                Console.WriteLine(string.Format("404 Error:{0}", Url1));
            }
        }

Download a file from a variable path every 15 minutes

3 Answers3