0

I have the following code:

 #!/usr/bin/env python
 import os
 os.system("wget -directory-prefix=myDir/1 URL1")
 os.system("wget -directory-prefix=myDir/2 URL2")
 os.system("wget -directory-prefix=myDir/3 URL3")

Between each call the script does "hang", when I press ctrl-c the script continues to work. Any ideas on how to troubleshoot this?

Edit: I now edited my question entirely.

Stephan Kristyn
  • 15,015
  • 14
  • 88
  • 147
  • 1
    Apologies if this is here, but I don't get it - what is your question? Do you want to do something different? If so, what do you want to do differently? – darthbith Jun 26 '14 at 12:01
  • Without having your certificate files or knowing what URLs you're trying to download, I can reproduce this exactly, but I tried running some basic `wget` commands in the same way and they ran serially. I did not try the `-r` option though, that could have something to do with it (perhaps the `-r` causes it to fork or something). What directories did it create, because according to what you posted they should all be saved to `myDir/1`? And what makes you think they are not running serially? – brianmearns Jun 26 '14 at 12:09
  • Did change my question, sorry for not being much clearer before. – Stephan Kristyn Jun 26 '14 at 12:59
  • This is working for me, each download is done serially and each file is saved in its directory – Luciano Afranllie Jun 26 '14 at 13:14

1 Answers1

3

Python is waiting for your files to download because os.system() is not asynchronous. As long as you don't have to actually interact with those files in the script, you can download them asynchronously with subprocess.Popen().

Check out this post for more info How can I run an external command asynchronously from Python?

Community
  • 1
  • 1
cephalopodMD
  • 169
  • 9
  • Well wouldn't that completely hog my hdd i/o as well as network bandwidth? I am downloading huge amounts of files and gigabytes. – Stephan Kristyn Jun 26 '14 at 14:07
  • 1
    One way or another, your system will be taking over if you want to use wget, and the way these download processes run will not be directly under your control. Popen will at least make the python process itself less screwy. However, if you want to be able to download these files serially and still have the whole process under the script's control (and avoid these weird hang issues), I'd suggest using a python library like urllib2 or something instead of hijacking os.system. – cephalopodMD Jun 26 '14 at 15:06
  • Your answer will help me, although I am still curious why my script (as above) does stop between the wget steps. When step 1 is finished, step 2 is NOT started, but I need to press ctrl-c for it to continue. – Stephan Kristyn Jun 26 '14 at 22:11