1

I download Data over a restAPI and wrote a module. The download takes lets say 10sec. During this time, the rest of the script in 'main' and in the module is not running until the download is finished. How can I change it, e.g. by processing it in another core?

I tried this code but it does not do the trick (same lag). Then I tried to implement this approach and it just gives me errors, as I suspect it 'map' does not work with 'wget.download'?

My code from the module:

from multiprocessing.dummy import Pool as ThreadPool
import urllib.parse

#define the needed data

function='TIME_SERIES_INTRADAY_EXTENDED'                  
symbol='IBM'                                              
interval='1min'                                            
slice='year1month1'                                        
adjusted='true'                                          
apikey= key[0].rstrip()

#create URL

SCHEME = os.environ.get("API_SCHEME", "https")
NETLOC = os.environ.get("API_NETLOC", "www.alphavantage.co")  #query?
PATH = os.environ.get("API_PATH","query")
query = urllib.parse.urlencode(dict(function=function, symbol=symbol, interval=interval, slice=slice, adjusted=adjusted, apikey=apikey))
url = urllib.parse.urlunsplit((SCHEME, NETLOC,PATH, query, ''))

#this is my original code to download the data (working but slow and stopping the rest of the script)

wget.download(url, 'C:\\Users\\x\\Desktop\\Tool\\RAWdata\\test.csv')

#this is my attempt to speed things up via multithreading from code

pool = ThreadPool(4)                             
if __name__ == '__main__':
    futures = []
    for x in range(1):
        futures.append(pool.apply_async(wget.download, url,'C:\\Users\\x\\Desktop\\Tool\\RAWdata\\test.csv']))
    # futures is now a list of 10 futures.
    for future in futures:
        print(future.get())

any suggestions or do you see the error i make?

Jackdaw
  • 7,626
  • 5
  • 15
  • 33
chubaka
  • 51
  • 8
  • so you're saying you want async requests? I'd look into splitting your code into async functions and using asyncio to run them asynchronously – bherbruck Feb 15 '21 at 18:09
  • not necessarily, i want to download and save a .csv without the rest of the code waiting until the download is completed. any way would do – chubaka Feb 16 '21 at 03:23

1 Answers1

2

ok, i figured it out, so i will leave it here in case someone else needs it.

  1. I made a module called APIcall which has a function APIcall() which uses wget.download() to download my data.

in main, i create a function (called threaded_APIfunc) which calls the APIcall() function in my modul APIcall

import threading                                                       
import APIcall  

def threaded_APIfunc():                                                      
APIcall.APIcall(function, symbol, interval, slice, adjusted, apikey)
print ("Data Download complete for ${}".format(symbol))

and then i run the threaded_APIfunc within a thread like so

threading.Thread(target=threaded_APIfunc).start()                            
print ('Start Downloading Data for ${}'.format(symbol))

what happends is, that the .csv file gets downloaded in the background, while the main loop doesent wait till the download ir completed, it does the code what comes after the threading right away

chubaka
  • 51
  • 8