0

I am required to download a .gz file (using GET) from a URL, uncompress it and then store it in S3.

I have written the following code to download file to a directory but I am struggling to uncompress it and store it in S3.

URL= https://api.botify.com/v1/jobs/

def download_file(url, folder_name):
    local_filename = url.split('/')[-1]
    path = os.path.join("{}\{}".format(folder_name, local_filename))
    with requests.get(url, stream=True) as r:
        with open(path, 'wb') as f:
            shutil.copyfileobj(r.raw, f)

How can I download the file from the given URL and unzip and store in S3 ?

Can someone please help.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
marie20
  • 723
  • 11
  • 30
  • Does this answer your question? [How to unzip gz file using Python](https://stackoverflow.com/questions/31028815/how-to-unzip-gz-file-using-python) – luk2302 May 26 '23 at 11:18
  • To upload files to S3, see: [Uploading files - Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html) – John Rotenstein May 26 '23 at 12:04

0 Answers0