5

I am trying to use Python's Popen to change my working directory and execute a command. 

pg = subprocess.Popen("cd c:/mydirectory ; ./runExecutable.exe --help", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
buff,buffErr = pg.communicate()

However, powershell returns "The system cannot find the path specified." The path does exist.

If I run

 pg = subprocess.Popen("cd c:/mydirectory ;", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)

it returns the same thing.

However, if i run this: (without the semicolon)

pg = subprocess.Popen("cd c:/mydirectory",stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)

The command returns without an error. This leads me to believe that the semicolon is issue. What is the cause for this behavior and how can I get around it?

I know I can just do c:/mydirectory/runExecutable.exe --help, but I would like to know why this is happening.

UPDATE :

I have tested passing the path to powershell as the argument for Popen's executable parameter. Just powershell.exe may not be enough. To find the true absolute path of powershell, execute where.exe powershell. Then you can pass it into Popen. Note that shell is still true. It will use the default shell but pass the command to powershell.exe

powershell = C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
pg = subprocess.Popen("cd c:/mydirectory ; ./runExecutable.exe", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True, executable=powershell)
buff,buffErr = pg.communicate()
//It works!
Jay S.
  • 1,318
  • 11
  • 29

2 Answers2

7

In your subprocess.Popen() call, shell=True means that the platform's default shell should be used.

While the Windows world is - commendably - moving from CMD (cmd.exe) to PowerShell, Python determines what shell to invoke based on the COMSPEC environment variable, which still points to cmd.exe, even in the latest W10 update that has moved toward PowerShell in terms of what the GUI offers as the default shell.

For backward compatibility, this will not change anytime soon, and will possibly never change.

Therefore, your choices are:

  • Use cmd syntax, as suggested in Maurice Meyer's answer.

  • Do not use shell = True and invoke powershell.exe explicitly - see below.

  • Windows only: Redefine environment variable COMSPEC before using shell = True - see below.

A simple Python example of how to invoke the powershell binary directly, with command-line switches followed by a single string containing the PowerShell source code to execute:

import subprocess

args = 'powershell', '-noprofile', '-command', 'set-location /; $pwd'
subprocess.Popen(args)

Note that I've deliberately used powershell instead of powershell.exe, because that opens up the possibility of the command working on Unix platforms too, once PowerShell Core is released.

Windows only: An example with shell = True, after redefining environment variable COMSPEC to point to PowerShell first:

import os, subprocess    

os.environ["COMSPEC"] = 'powershell'

subprocess.Popen('Set-Location /; $pwd', shell=True)

Note:

  • COMSPEC is only consulted on Windows; on Unix platforms, the shell executable is invariably /bin/sh

  • As of Windows PowerShell v5.1 / PowerShell Core v6-beta.3, invoking powershell with just -c (interpreted as -Command) still loads the profiles files by default, which can have unexpected side effects (with the explicit invocation of powershell used above, -noprofile suppresses that).

    • Changing the default behavior to not loading the profiles is the subject of this GitHub issue, in an effort to align PowerShell's CLI with that of POSIX-like shells.
mklement0
  • 382,024
  • 64
  • 607
  • 775
  • I'm trying to use *powershell* as the default python OS shell, but it always load the profile. Is there a work around how to do both; using python's `executable=` **and** using the `-NoProfile` argument? As an option, perhaps one can manipulate the env variable `ComSpec`? But IDK if that helps, because maybe then the entire environment is lost? – not2qubit May 01 '20 at 19:33
  • @not2qubit `ComSpec` is expected to contain an executable-file path only, so you wouldn't be able to specify `-NoProfile`. Not sure what you mean by environment being lost. – mklement0 May 01 '20 at 20:17
  • Aha, ok. I meant that modding ComSpec might influence subsequent calls. I.e. If I set ComSpec somewhere in Py script, then any subsequent os calls would still have that, or does it get reset upon a new `subprocess.Popen()` call, when specifying a different `executable`? (I guess it doesn't.) BTW. I am asking all this, because of [this](https://stackoverflow.com/a/61469226/1147688). – not2qubit May 01 '20 at 20:30
  • 1
    @not2qubit Yes, any subsequent `subprocess.Popen()` call with `shell = true` will be affected and more generally, any child process will see the modified `ComSpec` value. You could localize the modification (restore the original value after a call), but that seems hardly worth it (and it assumes that no child-process-creating code runs in parallel while the modified value is in effect). – mklement0 May 01 '20 at 20:36
0

You can concat multiple commands using '&' character instead of a semicolon. Try this:

pg = subprocess.Popen("cd c:/mydirectory & ./runExecutable.exe --help", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
buff,buffErr = pg.communicate()
Maurice Meyer
  • 17,279
  • 4
  • 30
  • 47