How to run shell commands inside python

Question

I need to create AWS lambda function to execute the python program. I need to incorporate the following shell command in it.

curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select(.region=="ap-southeast-1") | .ip_prefix'

Could someone guide me on this.

Does this help ? https://stackoverflow.com/questions/89228/how-to-execute-a-program-or-call-a-system-command-from-python Same here: https://stackoverflow.com/questions/14457756/how-to-run-commands-on-shell-through-python — kakou, Jun 14 '21 at 08:29
This smells like an X/Y problem – you really shouldn't shell out to `curl` and `jq` to download and parse JSON. — AKX, Jun 14 '21 at 08:30

score 1 · Accepted Answer · answered Jun 14 '21 at 08:34

1

To simply shell out to curl and jq to get that data,

import subprocess

data = subprocess.check_output("""curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select(.region=="ap-southeast-1") | .ip_prefix'""", shell=True)

but you really probably shouldn't do that since e.g. there's no guarantee you have curl and jq in the Lambda execution environment (not to mention the overhead).

Instead, if you have the requests library,

import requests

resp = requests.get("https://ip-ranges.amazonaws.com/ip-ranges.json")
resp.raise_for_status()
prefixes = {
    r["ip_prefix"]
    for r in resp.json()["prefixes"]
    if r["region"] == "ap-southeast-1"
}

answered Jun 14 '21 at 08:34

AKX

152,115
15
115
172

Many thanks. Is their a way for me to store all the IPs in a single file – Vidarbha Yalagadde Jun 14 '21 at 09:12
In general, yes, of course you can write them into a file. However, you can't write files in lambda functions. – AKX Jun 14 '21 at 09:14
Understood. I need to fetch these output IPs and store them in a file, later dump the file to S3 for further processing. – Vidarbha Yalagadde Jun 14 '21 at 09:25
So you need to store them in S3? Fine, that you _can_ do; the `boto3` library is the go-to S3 client for Python. – AKX Jun 14 '21 at 09:27

How to run shell commands inside python

1 Answers1