How can I safely escape a shell command with lots of quotes as a Python string?

Question

I have the following awk command:

 awk FPAT="([^,]+)|(\"[^\"]+\")" '{ print  $7048 }' ...

I want to submit it using os.system(cmd), but I cannot find the right way to write the code in a python string with all the special characters. The following is my latest try:

 cmd = " awk -v  FPAT=\"([^,]+)|(\\""[^\\""]+\\"")""  ' {{ print   %s  }} '  %s  > %s" .format(c,input,output)

and I get the following result:

' awk -v  FPAT="([^,]+)|(\\[^\\]+\\)  \' { print   %s  } \'  %s  > %s'

Can someone please help me?

I'd use a raw longstring. [Example](https://ideone.com/dbddiv) — Aaron, Aug 10 '20 at 19:09
BTW, you can also use raw-escaped triple-quotes. `cmd = r''' ...command here... '''` — Charles Duffy, Aug 10 '20 at 19:09

score 2 · Answer 1 · answered Aug 10 '20 at 19:14

You are confusing the placeholders used by the % operator with the field specifiers used by the format method.

cmd = "awk -v  FPAT=\"([^,]+)|(\\""[^\\""]+\\"")""  ' {{ print   {}  }} '  {}  > {}" .format(c, input, output)

Further, you should use the subprocess module instead of os.system. Python can handle the output redirection, so you don't need a shell.

with open(output, "w") as f:
    subprocess.run(["awk", "-v", 'FPAT=([^,]+)|("[^\\"]+")', '{{ print {} }}'.format(c), input])

Assuming Python 3.6 or later, you can simplify the awk script using an f-string literal.

with open(output, "w") as f:
    subprocess.run(["awk", "-v", 'FPAT=([^,]+)|("[^\\"]+")', f'{{ print {c} }}', input])

wjandrea · Accepted Answer · 2020-08-10T19:28:10.740

1

You can use input() to conveniently convert text into a string that you can copy-paste into your source code:

>>> cmd = input()
 awk FPAT="([^,]+)|(\"[^\"]+\")" '{ print  $7048 }' ...
>>> cmd
' awk FPAT="([^,]+)|(\\"[^\\"]+\\")" \'{ print  $7048 }\' ...'

In Python 2, use raw_input() instead.

BTW, avoid using input as a variable name since it shadows the builtin input(), and would make this method not work.

Or, if the text doesn't contain any triple-quotes, you can use a triple-quoted raw string:

>>> cmd = r''' awk FPAT="([^,]+)|(\"[^\"]+\")" '{ print  $7048 }' ...'''
>>> cmd
' awk FPAT="([^,]+)|(\\"[^\\"]+\\")" \'{ print  $7048 }\' ...'

_{Credit to Charles Duffy for suggesting the raw string option in a comment.}

edited Aug 10 '20 at 19:28

answered Aug 10 '20 at 19:07

wjandrea

28,235
9
60
81

Do we really care about Python 2 anymore, unless specifically mentioned by the OP? – chepner Aug 10 '20 at 19:14
That said, I don't think `input` is necessary for the question; it's about how to format a string suitable for use with `os.system`, not how to input a string from standard input. – chepner Aug 10 '20 at 19:15
@chepner That's why I didn't mention it in the first place. But since it's such a minor addition, and could be super confusing to a newbie, I added it. – wjandrea Aug 10 '20 at 19:17
@chepner Oh yeah you're right, the question's a bit ambiguous. Where OP's double-quotes aren't properly escaped, I assumed that's what they're asking. I edited to clarify what I mean a bit. – wjandrea Aug 10 '20 at 19:18
i am confused. when you use re the triple-quoted raw string, why does the d output has 2 backslashes? – wwla Aug 11 '20 at 13:47
@Emile See [Why do backslashes appear twice?](https://stackoverflow.com/q/24085680/4518341) – wjandrea Aug 11 '20 at 13:49
@wjandrea Thanks! and sorry if it was a dumb question – wwla Aug 11 '20 at 14:57

How can I safely escape a shell command with lots of quotes as a Python string?

2 Answers2