0

First post.

I cannot get DownloadFileAsync to do ANYTHING or generate an exception. Also, plain jane DownloadFile produces a local file but it is empty. I suspect the problem lies with the site serving up the response but it is really annoying that WebClient seems completely oblivious that nothing happens...

The source file is here: https://www.sec.gov/Archives/edgar/daily-index/xbrl/companyfacts.zip

It is a rather large file ~800MB. If I drop the url in the browser, it goes into download directly and finishes in ~ 5 minutes.

Running Win10 with .NET Framework 4.7.2.

Any insight you can offer would be awesome.

'Already consulted all these releated posts: 
'https://stackoverflow.com/questions/39292147/how-can-i-download-a-zip-file-from-a-url-using-c
'https://stackoverflow.com/questions/54497601/how-do-i-show-progress-bar-when-downloading-a-file
'https://stackoverflow.com/questions/153451/how-to-check-if-system-net-webclient-downloaddata-Is-downloading-a-binary-file
'https://stackoverflow.com/questions/3272067/webclient-403-forbidden?rq=1
 
Public DownloadClient As WebClient

Public Sub DownloadAsync()
   ' this is an approximately 800MB file 
    Dim ArchiveUrl As String = "https://www.sec.gov/Archives/edgar/daily-index/xbrl/companyfacts.zip"
    Dim UserFolder As string = System.IO.Path.GetTempPath()
    Dim ArchiveTemp as String = Now().ToString("yy-MM-dd_HH-mm") & ".zip"

    If DownloadClient Is Nothing = False AndAlso DownloadClient.IsBusy Then
        Debug.WriteLine("File download in progress")
        Return
    End If

    Path = New FileInfo(System.IO.Path.Combine(UserFolder, ArchiveTemp))
    Debug.Print("Downloading to: " & Path.FullName)

    'For .net 4.
    'ServicePointManager.SecurityProtocol = DirectCast(3072, SecurityProtocolType)

    'For .net4.5 Or later
    ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12

    DownloadClient = New WebClient()
    'DownloadClient.Headers.Add("User-Agent", "Mozilla/4.0 (compatible; MSIE 8.0)")
    'DownloadClient.Headers.Add("user-agent", " Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0")
    DownloadClient.Headers.Add("User-Agent: Other")

    AddHandler DownloadClient.DownloadDataCompleted, AddressOf DownloadComplete
    AddHandler DownloadClient.DownloadProgressChanged, AddressOf DownloadProgress

    Try
        DownloadClient.DownloadFileAsync(New Uri(ArchiveUrl), Path.FullName)
    Catch ex As WebException
        ' never throws an exception 
        If ex.Response Is Nothing = False Then
            Dim dataStream As Stream = ex.Response.GetResponseStream
            Dim reader As New StreamReader(dataStream)
            Debug.WriteLine(reader.ReadToEnd())
        Else
            Debug.WriteLine("No Response Stream")
        End If
    End Try

    'Also tried this - produces a zero size file - no errors 
    'DownloadClient.DownloadFile(New Uri(ArchiveUrl), Path.FullName)

End Sub
    
    
Public Sub DownloadProgress(sender As Object, e As DownloadProgressChangedEventArgs)
    Debug.WriteLine("Downloading: " & e.ProgressPercentage.toString)
End Sub

Public Sub DownloadComplete(sender As Object, e As System.ComponentModel.AsyncCompletedEventArgs)
    If e.Error Is Nothing = False Then
        Debug.WriteLine(e.Error.InnerException.ToString)
        Return
    End If

    Threading.Thread.Sleep(3000)
    Debug.WriteLine ("Download Complete!") 
End Sub
Berries
  • 45
  • 5
  • "my browser works but my code doesn't" is usually resolved by making your code look more like a browser: Press F12, request the file, inspect the headers the browser sent and send the same headers from your code. I see you've had a go at sending some particularly ancient useragent headers - send the actual one that works for your browser (which is unlikely to be IE8) - the remote server probably thinks you're a bot, and not sending 800mb to you is hence wise – Caius Jard Sep 12 '21 at 17:28
  • ps; do you really want to dump 800mb into the vs debug window? – Caius Jard Sep 12 '21 at 17:31
  • Try `DownloadFileTaskAsync()` and `Await` it – Charlieface Sep 12 '21 at 18:29
  • That site has an explicit security measure that limits automatic downloads. It requires a response from the request agent. -- Use a real WebBrowser to download that file. – Jimi Sep 12 '21 at 18:44

1 Answers1

0

I tried this and it worked

Using wc As New Net.WebClient
       wc.Headers.Add("User-Agent", "My Edgar Reader Admin@Example.com")
       wc.DownloadFile("https://www.sec.gov/Archives/edgar/daily-index/xbrl/companyfacts.zip", IO.Path.Combine(Environment.CurrentDirectory, "file.zip"))
End Using
MsgBox("Done")

I used the User-Agent you provided in another post

AdityaDees
  • 1,022
  • 18
  • 39
NawafAlKhaldi
  • 56
  • 4