python subprocess command with json in it ('u' prefix error)

Question

I am trying to spawn a subprocess that invokes a command with a json parameter that I get from a file. However, the json parameter used by the subprocess seems to produce an error because the json has a u prefix to indicate it is unicode.

I am not sure how I can get around this as I thought I had already converted the json to a string when I form the command

import sys
import json
from subprocess import Popen, PIPE

json_file = sys.argv[1]

# Read the provided policy file
with open(json_file, 'r') as f:
    values= json.load(f)

for value in values:
    command = ('''value definition create --name {} --rules '{}'
        ''').format(value['name'], value['definition'])

    p = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
    stdout, stderr = p.communicate()
    print(stdout)
    print(stderr)

The std error is outputting the following:

error: unrecognized arguments: values uif: {ufield: utype, ...

Here is the json file:

[  
    {  
        "name":"testValue",
        "definition":{  
            "if":{  
                "field":"type",
                "in":[  
                    "TestValue"
                ]
            },
            "then":{  
                "effect":"Deny"
            }
        }
    }
]

Please show what `values` and `command` are. – Alasdair Jun 19 '17 at 08:59 — Alasdair, Jun 19 '17 at 08:59

Jonathon Reinhart · Answer 1 · 2017-06-19T10:28:18.080

1

Command-line programs take byte strings, not unicode objects. Usually this means they accept ASCII strings, but it could be any encoding. You haven't indicated the encoding your program expects.

Try this: before format-ing a string to be passed to Popen call .encode('utf-8') on it.

Also, I see that definition is a complex object. When you pass it to str.format, it is going to be formatted as a string with Python syntax. It seems unlikely that this is actually what your external program expects.

It's always best to avoid shell=True unless you absolutely need it, particularly if your command-line arguments might have quotes in them. You should do this instead:

args = [
    'value',
    'definition',
    'create',
    '--name', value['name'],
    '--rules', value['definition'],
]
p = Popen(command, stdout=PIPE, stderr=PIPE)

edited Jun 19 '17 at 10:28

answered Jun 19 '17 at 09:13

Jonathon Reinhart

132,704
33
254
328

I tried calling .encode('utf-8') on the command before Popen, but sadly it had no effect as the u's are still present in the command – Pectus Excavatum Jun 19 '17 at 09:41
Also, yes I ideally need the output to be indented so its like it is in the file, just surrounded by ' ' but obviously without the u – Pectus Excavatum Jun 19 '17 at 09:49
Can you show a working example of calling this program? – Jonathon Reinhart Jun 19 '17 at 10:25
Perhaps this question will help you: https://stackoverflow.com/questions/956867/how-to-get-string-objects-instead-of-unicode-from-json-in-python – Jonathon Reinhart Jun 19 '17 at 10:28

python subprocess command with json in it ('u' prefix error)

1 Answers1