Python multiprocessing not executing per item but rather per character

Question

I have multiple environments (DEV, INT, PRD). I will like to build each environment asynchronously using python multiprocessing.

In my example below, build_environments is a list as such : ['DEV', 'UAT', 'PRD'].

def thread_set(self,environment):
        max_workers = 3
        concurrent = futures.ThreadPoolExecutor(max_workers)
        with concurrent as ex:
            ex.map(self.build, environment)
        return environment

def multibuild(self):
        build_environments = self.myparser.getEnvironments()     
        with multiprocessing.Pool(processes=4, maxtasksperchild=1) as pool:
             results = pool.imap(self.thread_set, build_environments)
             pool.close() 
             pool.join()

def build(self,build_environment):
        print(build_environment)

When i execute multibuild(), the print statement in build() function is printing and executing in this manner as shown below.

U
A
T
D
E
V
P
R
D

But I will want the build() print function to be :

DEV
UAT
PRD

What could be the issue ?

Both mapping functions require lists as arguments. The inner one only receives a string and splits it. — Klaus D., Apr 21 '20 at 16:26

dmmfll · Accepted Answer · 2020-04-24T19:34:39.757

4

Using print is not thread-safe. Use logging instead.

import logging

# environment is a string which is iterable, only one
# character is getting passed to build
def thread_set(self,environment):
    max_workers = 3
    concurrent = futures.ThreadPoolExecutor(max_workers)
    with concurrent as ex:
        ex.map(self.build, (environment,)) # place environment in a container, a list should work, too.
    return environment

# replace the print statement with logging

def build(self,build_environment):
    # print(build_environment) (Not thread-safe.)
    logging.info(build_environment)

Reference: An Intro to Threading in Python

edited Apr 24 '20 at 19:34

answered Apr 21 '20 at 16:22

dmmfll

2,666
2
35
41

thank you but this does not really answer the question. Printing was just an example, but i need to use the environment in that function. However instead of getting an environment, i get a letter. – Knitey Apr 21 '20 at 16:56
1

I made a change. Try that and see what happens. – dmmfll Apr 21 '20 at 17:32
In the future you may run into an issue where you need to pass more than one argument. See this answer that addresses that issue: https://stackoverflow.com/a/5442981/1913726 – dmmfll Apr 21 '20 at 17:55

Python multiprocessing not executing per item but rather per character

1 Answers1