progressbars inside multiprocessing calls overwriting each other on display

Question

I'm using progressbar 3.6.2, and getting incorrect display output when running multiple progressbars simultaneously. They write "on top" of each other.

Here's a minimum working example:

    import multiprocessing as mp
    import progressbar

    def run_many:
      p = mp.Pool(mp.cpu_count(), maxtasksperchild=1)
      results = p.map(run_one, args_list)
      p.close()
      p.join()

    def run_one:
       with progressbar.Progressbar(max_value=5000) as bar: 
           for i in range(5000):
               do_heavy_lifting(i)
               bar.update(i)

I get output that is one line long but oscillates between something that looks like

30% (1500 of 5000) |#######------------| Elapsed Time: 0:00:39 Time: 0:00:39

and

29% (1400 of 5000) |######-------------| Elapsed Time: 0:00:39 Time: 0:00:39

as each thread updates its own progress bar asynchronously. It looks like they're just displaying on top of each other.

Blindly inserting a line-break before going into the loop doesn't work (no surprise); what's a better option?

Answer 1

This is happening because more than 1 process is writing to the stdout, so they overwrite each other.

What you have to do is:

1. create a multiprocessing queue results = mp.Queue()
2. feed it results from within your child processes like results.put((mp.current_process(), status))
3. from the main process you have to read this data process_id, status = results.get()
4. display a progress bar for each process

Here is a perfect code that I based of when writing my multiprocessing progress monitor: Dynamic refresh printing of multiprocessing or multithreading in Python