Writing to file in a Popen subprocess

Question

I am attempting to log the simultaneous serial outputs of 3 GPS receivers to compare their performance on a single computer.

Following the structure outlined in this post, I created a main file which takes in inputs about the current trial for the file naming convention, then POpens a subprocess for each port:

import time
from datetime import datetime
from subprocess import Popen, PIPE


save_dir = "output_csvs/"

sparkfun_port = "COM7"
sparkfun_baud = "38400"

trimble_port = "COM6"
trimble_baud = "38400"

duro_port = "COM5"
duro_baud = "115200"

if __name__ == "__main__":
    # take input to generate file prefix
    file_prefix = "" + datetime.now().strftime("%m-%d-%Y-%H:%M:%S")
    # 180 for 3 min, 300 for 5 min
    trial_length = input("How long is this trial (in min)? ")
    # VSS6037L (Tallysman Surface Mount)
    # M8HCT (Maxtena)
    # VSP6037L (Tallysman Marine)
    # HC977XF (Tallysman helical)
    # GPS500 (Swift)
    # Zephyr (Trimble)
    antenna = input("Which GPS antenna is being used? ")
    file_prefix += "_" + antenna + trial_length + "min"

    # create filepath for each reciever
    sparkfun_path = save_dir + file_prefix + "_sparkfun.csv"
    trimble_path = save_dir + file_prefix + "_trimble.csv"
    duro_path = save_dir + file_prefix + "_duro.csv"

    # Popen subprocess for each reciever
    sparkfun = Popen(['python', './swap-c_ReadCOM.py', sparkfun_port, sparkfun_baud, sparkfun_path],
                     stdin=PIPE, stdout=PIPE, stderr=PIPE)
    trimble = Popen(['python', './swap-c_ReadCOM.py', trimble_port, trimble_baud, trimble_path],
                    stdin=PIPE, stdout=PIPE, stderr=PIPE)  
    duro = Popen(['python', './swap-c_ReadCOM.py', duro_port, duro_baud, duro_path], stdin=PIPE, stdout=PIPE,
                    stderr=PIPE)  

    # sleep for trial length (+ some margin to be trimmed) then close
    time.sleep(int(trial_length)*60+1)
    print("Trial Complete")
    quit()

Then, I created the subprocess file swap-c_ReadCOM.py, which is responsible for opening the specified COM port, listening to it, filtering for only GGA nmea strings, and writing said strings to a csv file.

swap-c_ReadCOM.py

import sys
import serial
import re
import csv


def trim_checksum(decoded_str):
    idx = decoded_str.find('*')
    if idx != -1:
        return decoded_str[:idx]
    return decoded_str


filepath = str(sys.argv[3])
ser = serial.Serial(port=sys.argv[1], baudrate=int(sys.argv[2]), bytesize=serial.EIGHTBITS, parity=serial.PARITY_NONE, stopbits=serial.STOPBITS_ONE)

while True:  # The program never ends... will be killed when master is over.
    ser_bytes = ser.readline()
    decoded_bytes = ser_bytes[0:len(ser_bytes) - 2].decode("utf-8")
    print(decoded_bytes)
    isGGA = re.search("\$\w\wGGA", decoded_bytes)
    if isGGA is not None:
        decoded_bytes = trim_checksum(decoded_bytes)
        with open(filepath, "a", newline='') as f:
            split = decoded_bytes.split(",")
            writer = csv.writer(f)
            writer.writerow(split)

To test this code, I began by attempting to only run one subprocess, commenting out the others. The main file runs to completion, but no csv is generated. Should I be piping the serial input back to main, to write to a file from there?

After modifying the main to print the output of my subprocess, it seems that swap-c_ReadCOM.py doesn't capture the serial input when run as a subprocess, as my code simply printed b' '. The csv file wouldn't be created then, as the regex is never cleared. When running swap-c_ReadCOM.py from the command line, the serial input is captured correctly and the csv file is created.

Tangentially, the code you put inside `if __name__ == '__main__':` should be absolutely trivial. The condition is only useful when you `import` this code; if all the useful functionality is excluded when you `import`, you will never want to do that anyway. See also https://stackoverflow.com/a/69778466/874188 — tripleee, Jul 11 '23 at 16:50
What platform are you running this on? You've got colons in your generated filenames, which aren't valid on all platforms. — jasonharper, Jul 11 '23 at 23:47
@jasonharper, that's a [good point](https://stackoverflow.com/questions/10386344/how-to-get-a-file-in-windows-with-a-colon-in-the-filename)! Thank you. Also, because "COM7", I think it's [Windows](https://stackoverflow.com/a/44056921/20307768). — Constantin Hong, Jul 12 '23 at 00:13
@jasonharper, I tested `with open('colon:colon.txt, 'a')` with a string to write. It created empty `colon` file in Windows 10. Interesting, I want to stay away from windows. — Constantin Hong, Jul 12 '23 at 00:27

Constantin Hong · Answer 1 · 2023-07-12T10:11:14.397

I can think of two possible explanations.

I created mock scripts called main.py and worker.py. Notice save_dir = "A_Place_With_No_Name/".

This code(main.py) run but doesn't create files. Because save_dir and ./worker.py are relative paths, your code won't write a file when you run the program in another directory or the wrong directory name of save_dir.

Also, in this case, your subprocesses died early because of FileNotFoundError: [Errno 2] No such file or directory:

So, change save_dir = "output_csvs/" and ./swap-c_ReadCOM.py to an absolute path, or if there is no save_dir in the directory, create it.

If your system is Windows, according to @jasonharper, you can't create file name with colon(file_prefix = "" + datetime.now().strftime("%m-%d-%Y-%H:%M:%S")). e.g., I tested with open('colon:colon.txt, 'a') with a string to write. It created empty colon file in Windows 10. So avoid ":" in file_prefix.

main.py

import time
from datetime import datetime
from subprocess import Popen, PIPE

# This is proof that the subprocess dies early without writing files.
save_dir = "A_Place_With_No_Name/"

sparkfun_port = "COM7"
sparkfun_baud = "38400"

trimble_port = "COM6"
trimble_baud = "38400"

duro_port = "COM5"
duro_baud = "115200"

if __name__ == "__main__":

    file_prefix = "" + datetime.now().strftime("%m-%d-%Y-%H:%M:%S")
    trial_length = input("How long is this trial (in min)? ")
    antenna = input("Which GPS antenna is being used? ")
    file_prefix += "_" + antenna + trial_length + "min"

    sparkfun_path = save_dir + file_prefix + "_sparkfun.csv"
    trimble_path = save_dir + file_prefix + "_trimble.csv"
    duro_path = save_dir + file_prefix + "_duro.csv"

    # Popen subprocess for each reciever
    sparkfun = Popen(['python', './worker.py', sparkfun_port, sparkfun_path],
                     stdin=PIPE, stdout=PIPE, stderr=PIPE)
    trimble = Popen(['python', './worker.py', trimble_port, trimble_path],
                    stdin=PIPE, stdout=PIPE, stderr=PIPE)  
    duro = Popen(['python', './worker.py', duro_port, duro_path], stdin=PIPE, stdout=PIPE,
                    stderr=PIPE)  

    # sleep for trial length (+ some margin to be trimmed) then close
    time.sleep(int(trial_length)*60+1)
    print("Trial Complete")
    quit()

worker.py

import sys
import time
import re
import csv


filepath = str(sys.argv[2])

while True:  # The program never ends... will be killed when master is over.
    print(sys.argv[1:])
    # FileNotFoundError: [Errno 2] No such file or directory:
    with open(filepath, "a", newline='') as f:
        split = sys.argv[1:]
        writer = csv.writer(f)
        writer.writerow(split)
    time.sleep(1)

Thank you for your feedback. Per number 1, I've replaced the path to the save directory and the path to the subprocess script with an absolute path, and ensured that the output_csvs subdirectory exists. Per number 2, I removed the colons from the datetime.now() call. Unfortunately, `swap-c_ReadCOM.py` still does not save the csvs. — Mattman7306, Jul 12 '23 at 12:37
@Mattman7306 How did you run `swap-c_ReadCOM.py` individually? Do they run with the same Python version? Because `python` and `python3` can differ. Please post the result of `python -V` and `python3 -V` too. — Constantin Hong, Jul 12 '23 at 13:46
Try debugging like [this](https://stackoverflow.com/a/3384209/20307768) but replace the exception statement with `except Exception as e:` and `with open(filepath + '-log', 'a'):`. — Constantin Hong, Jul 12 '23 at 15:14
@Constanin Hong I ran `swap-c_ReadCOM.py` using this terminal command 'python swap-c_ReadCOM.py COM7 38400 test.csv ` As I am using windows, the python 3 command does not exist. Wrapping the try except block around the serial port opening code and the writing to a csv code did not produce anything in the log file. It was also not created. — Mattman7306, Jul 13 '23 at 14:25
@Mattman7306, remove `stdin=PIPE, stdout=PIPE, stderr=PIPE` in Popen statements. With those options in `Popen`, you can't get any error messages. Also, Try `from subprocess import check_call, PIPE` instead of `Popen` and replace `Popen` with `check_call`. Let see what causes the problem. — Constantin Hong, Jul 13 '23 at 19:18
After making those changes, I see that the subprocess is not able to find the pyserial module `ModuleNotFoundError: No module named 'serial'` even though pyserial is installed in my environment and I am able to run `swap-c_ReadCOM.py` from the terminal without issue. — Mattman7306, Jul 14 '23 at 18:29
Okay, now then, replace `'python'` in `Popen`(or `check_call`) with `sys.executable`(this object is a string). e.g.: `check_call([sys.executable, './swap-c_ReadCOM.py', ....])`. If it works, debug is done. Just change only `'python'` in `Popen` with `sys.executable` in your original code but keep things like `Popen, stdin=PIPE`, and so on. — Constantin Hong, Jul 14 '23 at 20:03

Writing to file in a Popen subprocess

swap-c_ReadCOM.py

1 Answers1

main.py

worker.py