Save output of four parallely running python program to different log files

Question

I have four different programs running which is started using a single python program.

import os                                                                       
from multiprocessing import Pool 
import sys   
processes = ('p1.py', 'p2.py', 'p3.py','p4.py')
def run_process(process):                                                             
    os.system('python3 {}'.format(process)) 
pool = Pool(processes=4)                                                        
pool.map(run_process, processes)

currently iam getting the log of all programs in a single file using nohup pmain.py>test.log But how can i get four different logs of p1,p2,p3 and p4 respectively in different log files.

A simple fix would be to `processes = ('p1.py > p1.log', 'p2.py > p2.log', 'p3.py > p3.log', 'p4.py > p4.log')`. But don't do that. And don't do `os.system` as well — han solo, Dec 31 '19 at 04:28
Does this answer your question? [Log output of multiprocessing.Process](https://stackoverflow.com/questions/1501651/log-output-of-multiprocessing-process) — abhilb, Dec 31 '19 at 04:31
https://docs.python.org/3/howto/logging-cookbook.html#using-a-rotator-and-namer-to-customize-log-rotation-processing ,,, https://docs.python.org/3/library/logging.handlers.html#logging.handlers.QueueHandler — wwii, Dec 31 '19 at 04:49
Or this?..[How should I log while using multiprocessing in Python?](https://stackoverflow.com/questions/641420/how-should-i-log-while-using-multiprocessing-in-python) — wwii, Dec 31 '19 at 04:53

han solo · Accepted Answer · 2019-12-31T04:35:51.597

4

A simple fix will be to do,

import os                                                                       
from multiprocessing import Pool 
import sys   
processes = ('p1.py > p1.log', 'p2.py > p2.log', 'p3.py > p3.log','p4.py > p4.log')
def run_process(process):                                                             
    os.system('python3 {}'.format(process)) 
pool = Pool(processes=4)                                                        
pool.map(run_process, processes)

But don't do the above.

The right way to do that would be using subprocess with something like,

import subprocess
processes = ('p1.py', 'p2.py', 'p3.py','p4.py')
procs = []
for pname in processes:
  logfile = os.path.splitext(pname)[0] + '.log'
  with open(logfile, 'w') as f:
    proc = subprocess.Popen(['python3', pname], stdout=f)
    procs.append(proc)

for proc in procs:
  proc.wait()

edited Dec 31 '19 at 04:35

answered Dec 31 '19 at 04:29

han solo

6,390
1
15
19

why do you prefer the second way instead of the first method. – sharath Dec 31 '19 at 04:41
Because it is frowned upon to directly use `os.system`, and the second method is very readable, and does the same thing – han solo Dec 31 '19 at 04:45
`subprocess.Popen` is a non-blocking call. One would want to wait for all the processes to exit ? – han solo Dec 31 '19 at 04:48
No problem. I am glad, i was able to help – han solo Dec 31 '19 at 04:50

Save output of four parallely running python program to different log files

1 Answers1