114

I use Powershell's Invoke-WebRequest method to download a file from Amazon S3 to my Windows EC2 instance.

If I download the file using Chrome, I am able to download a 200 MB file in 5 seconds. The same download in PowerShell using Invoke-WebRequest takes up to 5 minutes.

Why is using Invoke-WebRequest slower and is there a way to download at full speed in a PowerShell script?

Lloyd Banks
  • 35,740
  • 58
  • 156
  • 248

6 Answers6

208

Without switching away from Invoke-WebRequest, turning off the progress bar did it for me. I found the answer from this thread: https://github.com/PowerShell/PowerShell/issues/2138 (jasongin commented on Oct 3, 2016)

$ProgressPreference = 'SilentlyContinue'
Invoke-WebRequest <params>

For my 5MB file on localhost, the download time went from 30s to 250ms.

Note that to get the progress bar back in the active shell, you need to call $ProgressPreference = 'Continue'.

Martin Prikryl
  • 188,800
  • 56
  • 490
  • 992
TinyTheBrontosaurus
  • 4,010
  • 6
  • 21
  • 34
  • 2
    This makes a huge difference for us too (tens of seconds down to less than a second for 10-30MB files) - I always had a suspicion that it was the progress display slowing things down but didn't know how to stop it doing it. – Mark Hughes Dec 16 '17 at 12:20
  • 4
    For a 100MB file this reduces the time from 10 Minutes to 2 Seconds. I wish developers would think more about the architecture of their software. – lanoxx Oct 22 '18 at 14:42
  • 48
    A lesson on how not to implement a progress bar. – sakra Jan 09 '19 at 19:56
  • 3
    For my 280MB file, it went from 33 minutes to 28 seconds. Crazy! – Danny Tuppeny Aug 07 '19 at 17:23
  • 29
    It appear the progress bar updates after every byte, which is utter madness. – OrangeDog Aug 22 '19 at 15:00
  • 12
    Can't believe that `-ProgressPreference` isn't a parameter with `Invoke-WebRequest` – CJBS May 05 '20 at 22:27
  • Updating after only every byte? I want real-time darnit, I want per bit progress bar! – jjxtra Aug 19 '22 at 17:17
  • 3
    Windows hasn't changed one bit. – nhooyr Dec 14 '22 at 11:19
  • 2
    Still a major issue in 2023. Note, this has [additional benefits](https://stackoverflow.com/q/75168064/3196753) as well. – tresf Jan 19 '23 at 19:28
  • 1
    I wasn't sure if this would make a difference if PowerShell is running non-interactively, but it does! I was using PowerShell to download a large file as part of a Jenkins pipeline, and this cut it down from over an hour to about 30 seconds – Aaron Christiansen May 29 '23 at 14:14
65

I was using

Invoke-WebRequest $video_url -OutFile $local_video_url

I changed the above to 

$wc = New-Object net.webclient
$wc.Downloadfile($video_url, $local_video_url)

This restored the download speed to what I was seeing in my browsers.

Lloyd Banks
  • 35,740
  • 58
  • 156
  • 248
  • 3
    So I just ran `wc.downloadFile(ibm-s3-url, "./test.tar.gz")`, it did something, presumably download that file, _but it didnt put it in my working directory_... any idea where it might have gone? – Groostav Aug 28 '19 at 23:51
  • @Groostav Mine showed up in c:\windows\system32. Not exactly the first place I'd look when downloading to a relative location. – BryanC Oct 13 '19 at 09:39
  • 1
    @Groostav the current process' working directory is *not* the same thing as PowerShell's current location. – Mark Oct 15 '21 at 23:38
23

$ProgressPreference = 'SilentlyContinue' I got this down from 52min down to 14sec, for a file of 450 M. Spectacular.

danimal
  • 1,567
  • 13
  • 16
njefsky
  • 327
  • 2
  • 5
  • 1
    please include more information in your answer, eg where does the setting you describe get set/defined? – danimal Aug 07 '20 at 14:49
  • 1
    This is a built-in [preference variable](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_preference_variables?view=powershell-5.1#progresspreference) to PowerShell, and controls whether or not a progress bar is displayed for certain operations, such as downloading files via `Invoke-WebRequest`. The reason this improves performance here is because `Invoke-WebRequest` (and `Invoke-RestMethod`) count the downloaded bytes too often (*every single byte* I believe), so the cmdlet is actually slowed down as it tallies how many bytes have been processed. – codewario Nov 06 '20 at 20:45
17

One-liner to download a file to the temp directory:

(New-Object Net.WebClient).DownloadFile("https://www.google.com", "$env:temp\index.html")
BlueSky
  • 1,449
  • 1
  • 16
  • 22
  • How does this answer the question? – Martin Prikryl Nov 12 '19 at 06:33
  • 4
    Lloyd asked: "is there a way to download at full speed in a PowerShell script?" This answer is a one-liner way to do that. Downloading to the temp directory is the canonical way to demonstrate this. – BlueSky Nov 13 '19 at 23:59
  • `Invoke-WebRequest`, which the OP is already using, does internally the same what `WebClient.DownloadFile`. – Martin Prikryl Nov 14 '19 at 06:20
  • 4
    This is actually a good answer, WebClient.DownloadFile is much faster than Invoke-WebRequest due to the way Invoke-WebRequest tracks its own progress – codeulike Nov 14 '20 at 14:30
  • 2
    Absolutely relevant, this fixes exactly the problem OP has (and mine :) ) – fl0w Feb 10 '21 at 09:45
  • I actually prefer this one, because you don't have to worry about clobbering `$ProgressPreference` or saving/restoring it. As long as you're simply downloading a file, I prefer the one-liner. – Phil Hamer Feb 12 '21 at 20:58
1

I just hit this issue today, if you change the ContentType argument to application/octet-stream it is much faster (as fast as using webclient). The reason is because the Invoke-Request command will not try and parse the response as JSON or XML.

Invoke-RestMethod -ContentType "application/octet-stream" -Uri $video_url  -OutFile $local_video_url
user1875674
  • 37
  • 2
  • 3
    `-ContentType` is for `POST` requests only AFAIK so it probably won't make a difference. – rednoah Nov 02 '16 at 19:30
  • 3
    You'd be better off using the `-UseBasicParsing` parameter but that won't make much of a difference for static files. Setting `$ProgressPreference = 'SilentlyContinue'` before calling `Invoke-WebRequest` ***greatly*** improves the performance. – codewario Jan 28 '20 at 17:58
0

Unfortunately the progress bar of Invoke-WebRequest is slowing file download a lot on Windows Powershell 5.1 (the version included in Windows OS). It's much faster on later Powershell versions (I tested it on Powershell 7.3).

IMO, if you are forced to use Windows Powershell then the best way is to use curl since it's included on Windows by default now. Just be aware that by default Windows Powershell has alias named curl for Invoke-WebRequest so to run curl program you need to use curl.exe to tell Windows Powershell that you don't want to use curl alias.

This command takes 11 minutes on Windows Powershell 5.1 and 23 seconds on Powershell 7.3:

Invoke-WebRequest -Verbose -Uri "https://download.visualstudio.microsoft.com/download/pr/7c048383-52b1-47cb-91d1-acfaf1a3fcc9/ea510c0bfa44f33cc3ddea79090a51e1/dotnet-sdk-6.0.410-win-x64.exe" -OutFile ".\dotnet-sdk-6.0.410-win-x64.exe"

and this takes 15 seconds:

curl.exe -fSLo .\dotnet-sdk-6.0.410-win-x64.exe https://download.visualstudio.microsoft.com/download/pr/7c048383-52b1-47cb-91d1-acfaf1a3fcc9/ea510c0bfa44f33cc3ddea79090a51e1/dotnet-sdk-6.0.410-win-x64.exe
Mariusz Pawelski
  • 25,983
  • 11
  • 67
  • 80