Is every instance of subprocess.Popen() its own shell?

Question

Background: I'm trying to make an application that plays music via a GUI (Tkinter), Youtube_DL and FFmpeg. While the actual application is done it also requires FFmpeg to be an environment variable to work. Now, I'm trying to foolproof the creation of a "personal" FFmpeg environment variable in order to make the application portable (the last step is using pyinstaller to make a .exe file).

Problem: I'm trying to create the environment variable for FFmpeg by passing SET through subprocess.Popen:

add_ffmpeg = subprocess.Popen(f"IF EXIST {path2set} SET PATH=%PATH%;{path2set}", shell=True)

When I try to echo %PATH% (with Popen) the FFmpeg variable that should be present, is not. I just need to know whether or not I'm wasting my time with SET and should instead be using SETX or perhaps some other solution, I'm open to being told I did this all wrong.

Relevant Code:

# get any sub-directory with ffmpeg in it's name
ffmpeg = glob(f"./*ffmpeg*/")

# proceed if there is a directory
if len(ffmpeg) > 0:
    # double-check directory exists
    ffmpeg_exist = path.isdir(ffmpeg[0])

    if ffmpeg_exist:
        print("FFmpeg: Found -> Setting Up")
        
        # get the absolute path of the directories bin folder ".\ffmpeg-release-essentials.zip\bin"
        path2set = f"{path.abspath(ffmpeg[0])}\\bin\\"
        
        # add path of directory to environment variables
        add_ffmpeg = subprocess.Popen(f"IF EXIST {path2set} SET PATH=%PATH%;{path2set}", shell=True)
        add_ffmpeg.wait()
        
        # print all of the current environment variables
        list_vars = subprocess.Popen("echo %PATH%", shell=True)
        list_vars.wait()

else:
    print("FFmpeg: Missing -> Wait for Download...")
    
    # download the ffmpeg file via direct link
    wget.download("https://www.gyan.dev/ffmpeg/builds/ffmpeg-release-essentials.zip")
    
    # unzip the file
    powershell = subprocess.Popen("powershell.exe Expand-Archive -Path './ffmpeg-release-essentials.zip' "
                                  "-DestinationPath './'")
    powershell.wait()

    # remove the file
    remove("./ffmpeg-release-essentials.zip")

    # get any sub-directory with ffmpeg in it's name
    ffmpeg = glob("./*ffmpeg*/")

    # double-check directory exists
    ffmpeg_exist = path.isdir(ffmpeg[0])
    
    # proceed with if it exists
    if ffmpeg_exist:
        print("FFmpeg: Found -> Setting Up")
        
        # get the absolute path of the directories bin folder ".\ffmpeg-release-essentials.zip\bin"
        path2set = f"{path.abspath(ffmpeg[0])}\\bin\\"
        
        # add path of directory to environment variables
        add_ffmpeg = subprocess.Popen(f"IF EXIST {path2set} SET PATH=%PATH%;{path2set}", shell=True)
        add_ffmpeg.wait()

        # print all of the current environment variables
        list_vars = subprocess.Popen("echo %PATH%", shell=True)
        list_vars.wait()
        
    else:
        print("Something unexplained has gone wrong.")
        exit(0)

But remember that all those subprocesses inherit YOUR environment. If you change `os.environ` to add or change variables, your subprocesses will see that. — Tim Roberts, Apr 26 '21 at 03:52
There is in real no need for `shell=True` to start with `subprocess.Popen()` first an instance of `cmd.exe` with option `/C` and the command line written in Python code appended as additional argument(s). It would be possible to run directly `ffmpeg.exe` with full qualified file name (drive + path + name + extension) without starting the Windows command processor at all. Please read the Python documentation for the [subprocess module](https://docs.python.org/3.9/library/subprocess.html) carefully and completely. — Mofi, Apr 26 '21 at 04:57
The description of the module and of all the parameters which can be passed to the `subprocess` functions can be understood better on reading the Microsoft documentation for the Windows kernel library function [CreateProcess](https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw) and the structure [STARTUPINFO](https://docs.microsoft.com/en-us/windows/desktop/api/processthreadsapi/ns-processthreadsapi-startupinfow) because of the `subprocess` module is on Windows just a Python wrapper for `CreateProcess` called without or with a `STARTUPINFO`. — Mofi, Apr 26 '21 at 05:02
There is one more common beginners mistake already in first line of posted code. The search for `ffmpeg` directory is done with a path relative to current directory. The current directory on starting `python.exe` to interpret the Python script or later the created executable can be any directory as an executable can be always started from any directory on using the full qualified file name of the executable. [sys.executable](https://docs.python.org/3/library/sys.html) should be used to get full path of the executable processing the Python script. That absolute path should be used finally. — Mofi, Apr 26 '21 at 05:10
See also [How do I get the path and name of the file that is currently executing?](https://stackoverflow.com/questions/50499/) `os.path.realpath(__file__)` can be used to get the absolute path of the Python script file currently interpreted by `python.exe` on coded Python script not yet converted to an executable. The directory with `ffmpeg` should be searched in directory returned by `os.path.realpath(__file__)` or of `sys.executable` if `ffmpeg` is always installed together with the Python script or an executable of it. — Mofi, Apr 26 '21 at 05:17
Please completely forget usage of `setx`. We really don't need one more application __corrupting__ our __user__ or __system__ `PATH` configuration just because of not knowing the limitations of `setx` and how this command really works and how it can be used at all. `setx` is very dangerous to use on modification of `PATH` as it truncates the string assigned to a persistent stored environment variable to 1024 characters. `PATH` can have multiple folder paths resulting in a string length of more than 1024 characters. `PATH` is not needed on using full qualified file names of executables. — Mofi, Apr 26 '21 at 05:22
An application designed for being a portable application should not modify the configuration of the machine on which the portable application is used at all as it would be done on using `setx` to modify __user__ `PATH`. So don't do that, but use Python code to get the absolute path of the Python script respectively the final executable and concatenate this path with the relative path to `ffmpeg` executable to get the full qualified file name of `ffmpeg` and use `subprocess` to run just `ffmpeg` with full qualified file name without usage of the shell interpreter (`cmd.exe` on Windows). — Mofi, Apr 26 '21 at 05:25
I took your advice and I completely removed messing with environment variables, instead opting to just use the ffplay exe directly. — saa-sof, Apr 26 '21 at 07:30

score 3 · Answer 1 · answered Apr 26 '21 at 05:13

Indeed, each subprocess is a separate process - that's what "subprocess" means - which means if you set something in one subprocess call, it will only be visible within that subprocess, and not in its parent process, or in other subprocesses you run subsequently.

A common solution is to pass an env keyword parameter to the subprocess call with any variables you want set;

from pathlib import Path
from os import environ

env = environ.copy()

if Path(path2set).exists():
    env['PATH'] = '%s;%s' % (path2set, env['PATH'])

subprocess.Popen([...], env=env)

Of course, in many cases, it suffices to perform this modification in the parent process, and have any subprocesses inherit the new value. (Just modify os.environment directly, instead of a copy, and remove the env=env from the last line).

Whether or not a subprocess is a shell is a separate topic, and really not relevant to your question. Trivially, with shell=True it is; but as in your example, if your subprocess is e.g. powershell.exe then it's a shell because that's what it is (and shell=True in addition is rather redundant, and probably something you want to avoid if possible).

Is every instance of subprocess.Popen() its own shell?

1 Answers1