xargs run bash script with threads (2 arguments)

Question

I have a file in the following format:

"192.168.2.1" "80"
"172.16.1.1" "443"
"10.1.1.1" "8080"

which contains multiple local IPs and i'm trying to run a script with threads using xargs to execute the following command:

cat local-IPs.txt|xargs -I % -P 100 python script.py % "execute command"

The problem I am facing right now is that since the local-IPs.txt file contains "IP" AND "port", the xargs command won't be able to take both (IP and PORT).

Any help would be very appreciated, thanks.

Kamil Maciorowski · Answer 1 · 2020-04-21T21:58:13.713

If you can, edit script.py so it takes IP and port as its last arguments. Then you can use the -n (or -L) option of xargs:

<local-IPs.txt xargs -n 2 -P 100 python script.py "execute command"

^{(Useless use of cat removed.)}

To be clear, it will execute something like

python script.py "execute command" 192.168.2.1 80

therefore it's crucial script.py expects this form. Even if you cannot modify script.py itself, maybe you can copy it and work with the copy.

Another approach is to make xargs spawn a shell whose only job will be to change the sequence of arguments. This should work with unmodified script.py:

<local-IPs.txt xargs -n 2 -P 100 sh -c 'exec python script.py "$1" "$2" "execute command"' xargs-sh

There will be one sh per python. This may substantially decrease the performance.

Note there are now double-quotes inside single-quotes. It's good to know how it affects expansion of parameters and variables, especially if execute command contains some (your comment indicates it doesn't, but in general it may).

xargs-sh is explained here.

xargs run bash script with threads (2 arguments)

1 Answers1