1

So I am trying to write a simple wrapper in python to call rasa, a nlu tool from. The command I would write on the the command line is this:

curl -X POST "localhost:5000/parse" -d '{"q":"I am looking for fucking Mexican food"}' | python -m json.tool

The output I expect is something like this:

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 545 0 500 100 45 33615 3025 --:--:-- --:--:-- --:--:-- 35714

plus the outprint of a json file.

I wrote this program in python: 

import subprocess

utterance = "Lets say something"


result = subprocess.run(["curl", "-X", "POST", "localhost:5000/parse", "-d", "'{\"q\":\""+utterance+"\"}'", "|", "python", "-m", "json.tool"], stdout=subprocess.PIPE)
print(vars(result))
print(result.stdout.decode('utf-8'))

Unfortunately my output is like this, meaning I dont actually get the return from the curl call:

{'args': ['curl', '-X', 'POST', 'localhost:5000/parse', '-d', '\'{"q":"Lets say something"}\'', '|', 'python', '-m', 'json.tool'], 'returncode': 2, 'stdout': b'', 'stderr': None}

If I call my python programm from the commandline, this is the output:

curl: option -m: expected a proper numerical parameter curl: try 'curl --help' or 'curl --manual' for more information {'args': ['curl', '-X', 'POST', 'localhost:5000/parse', '-d', '\'{"q":"Lets say something"}\'', '|', 'python', '-m', 'json.tool'], 'returncode': 2, 'stdout': b'', 'stderr': None}

I tried looking everywhere but just cant get it going. Would really appreciate some help.

Julian Kurz
  • 93
  • 1
  • 9
  • 2
    A thought, have you looked at pycurl or requests -- you would be writing native code, easier to implement, easier to read, easier to debug. – SteveJ Feb 27 '18 at 23:43

1 Answers1

1

Update: I grossly misunderstood the question the first time. Rushed reading the details, so my apologies there. You are having a problem because you are trying to pipe two commands together using Popen. The pipe operator, however, is something implemented by the shell, not python. So it is expecting your command to just be a command related to curl. It is complicated, but you have options.

I think for your particular example, the simplest is to not try to chain the command to json.tool. You actually have no need for it. You are already in python, so you can just pretty print the output you get from curl yourself. Using python would look something like

import json
import shlex
from subprocess import Popen, PIPE

command = 'curl -XGET http://localhost:9200'

p = Popen(shlex.split(command), stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate()

if p.returncode != 0:
    print(err)

j = json.loads(output.decode("utf-8"))
print(json.dumps(j, indent=4, sort_keys=True))

However, if what you want long term is to actually connect multiple processes with pipes, well there it depends on the scenario. The easiest method is to pass shell=True to Popen and pass the exact command (not a list of arguments). This delegates everything to the shell. I need to warn you that this is very exploitable when the command is based off user input. Both 2.x pipes.quote() and 3.x shlex.quote() have a recommendation of how to escape the command so it should be safe.

from subprocess import Popen, PIPE

command = 'curl -XGET http://localhost:9200 | python -m json.tool'

p = Popen(command, stdin=PIPE, stdout=PIPE, stderr=PIPE, shell=True)
output, err = p.communicate()

if p.returncode != 0:
    print(err)

print(output.decode("utf-8"))

So if you find yourself needing to connect processes but have something based on user input, you can use multiple processes and connect them yourself.

import shlex
from subprocess import Popen, PIPE

command1 = 'curl -XGET http://localhost:9200'
command2 = 'python -m json.tool'

p1 = Popen(shlex.split(command1), stdout=PIPE)
p2 = Popen(shlex.split(command2), stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()
output, err = p2.communicate()

if p2.returncode != 0:
    print(err)

print(output.decode("utf-8"))

This question has a bunch more on the topic if you are curious.

phospodka
  • 988
  • 1
  • 5
  • 12
  • I tried using Popen, unfortunatley your exact code with my command produces this: b"curl: option -m: expected a proper numerical parameter\ncurl: try 'curl --help' or 'curl --manual' for more information\n" b'' – Julian Kurz Feb 27 '18 at 22:31
  • 1
    I'll try something later, but you can probably drop the `| python -m json.tool` part. You are already in python so you can pretty print it in your code if need be. – phospodka Feb 27 '18 at 22:35
  • 1
    @JulianKurz updated the answer. I totally did not read your question close enough. What SteveJ said about pycurl also looks interesting as a nicer alternative to executing curl without invoking an external process. – phospodka Feb 28 '18 at 03:37
  • Thank you so much, just to learn something I tried both of your options, and they both work beautifully. So of course I implemented the first one. Although I might play around myself with trying to cause harm with the second option, since the user input is speech based and I find it interesting to see if it is still possible. – Julian Kurz Feb 28 '18 at 07:55
  • @JulianKurz I made one more update. Both 2.x and 3.x have a recommendation for escaping the command that I added mention of when using `shell=True`. I did not immediately get it to work so I'll try that later and update the example, but maybe worth trying. Turning text to speech into code is probably difficult, but still possible. If this helped accepting the answer would be super helpful. – phospodka Feb 28 '18 at 16:22
  • Looks like you maybe use those `quote()` methods on the strings used to construct the command, and not the whole command itself. – phospodka Feb 28 '18 at 16:28