Python process does not start without a delay

Question

I'm working on a complex application involving multiple sockets and threads/processes. For some reason, sometimes this particular process won't run unless a delay was invoked.

Something like:

class MyClass:
    def somefunc(self):
        some_thread = Thread(...)  # Some complex processing.
        some_thread.start()

        time.sleep(1)  # Some delay like this is needed. 

        some_process = Process(target=sometarget)
        print('Should start')
        some_process.start()
        print('Should have started')

    def sometarget(self):
        print('It started')

        # Do more complex processing here.

This prints out something like:

Should start
Should have started
It started  # This gets printed out only if there's a significant delay inserted

Not sure if it's just some random occurrence due to how Python schedules jobs underneath, but if I don't put a delay, it doesn't seem to run even after I wait a while.

*Something like:*? Your code, which is not even close to a [minimal, reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) can't possibly run because you need `some_process = Process(target=self.sometarget)`. Also, you have two completely irrelevant tags and missing one relevant tag, `multithreading`. But mostly: if your code can get to execute `time.sleep(1)` statement, it can probably get to execute the process-creation statements. So I don't believe you need the delay (**I didn't**). I could see a scenario where a delay might be needed -- but in the thread. — Booboo, Jun 03 '21 at 10:58

score 1 · Answer 1 · answered Jun 03 '21 at 04:46

1

When you use threading, you come up against the "global interpreter lock". The Python interpreter will only allow one thread at a time to be running Python code. Until you sleep or block for I/O, your main thread will keep running, and your threads will wait. Python threading is great for I/O-bound loads, but it is nearly useless for CPU-bound loads. That's why people use multiprocessing instead, where you're actually running a separate interpreter in a separate process.

answered Jun 03 '21 at 04:46

Tim Roberts

48,973
4
21
30

This is a great answer in general about the GIL and multithreading but I don't think it really explains what the OP observed and whose code clearly has an error and can't possibly run as is. – Booboo Jun 03 '21 at 11:21
it's not entirely true that the main thread will continue until it hits blocking i/o or sleep.. Threads enable true pre-emptive multi-tasking because they will release the GIL and allow other threads to have it for a while. By default each thread will release after about 5 miliseconds. This is a best effort algorithm that must switch in-between byte codes, so a long running process that amounts to a single bytecode in the interpreter may push the interval slightly longer. (see [this](https://stackoverflow.com/a/37911403/3220135) answer for more) – Aaron Jun 04 '21 at 00:27

Booboo · Answer 2 · 2021-06-03T11:23:44.210

See my comments to your question and Tim Roberts' answer concerning the Global Interpreter Lock (GIL). I could see a scenario where the started thread ("some complex processing") had a lock on the GIL and prevented the main thread from continuing to run and get a chance to start the process-creation statements. But it would probably not have a chance to execute the delay statement (time.sleep(1)) either. And if it did, what would that accomplish? If anything, that delay would need to be in the started thread so that it would release the GIL and allow the main thread to run. But I found that it was not necessary for that delay to be inserted. I assume that even though the thread is created and "started", the main thread continues to run and gets to create the new process before the new thread actually starts execution.

This is a complete, minimal, reproducible example:

from threading import Thread
from multiprocessing import Process
import time

def doWork():
    # 100% CPU-bound
    sum = 0
    for _ in range(100_000_000):
        sum += 1

class MyClass:
    def somefunc(self):
        some_thread = Thread(target=doWork)  # Some complex processing.
        some_thread.start()

        #time.sleep(1)  # Some delay like this is needed.

        some_process = Process(target=self.sometarget) #self.sometarget is needed
        print('Should start')
        some_process.start()
        print('Should have started')

    def sometarget(self):
        print('It started')

# this is required for Windows, which I have:
if __name__ == '__main__':
    mc = MyClass()
    mc.somefunc()

Prints:

Should start
Should have started
It started

Python process does not start without a delay

2 Answers2