How to use Queue with threading properly

Question

I am new to queue & threads kindly help with the below code , here I am trying to execute the function hd , I need to run the function multiple times but only after a single run has been completed

import queue
import threading
import time

fifo_queue = queue.Queue()

def hd():
    print("hi")
    time.sleep(1)
    print("done")


for i in range(3):
    cc = threading.Thread(target=hd)
    fifo_queue.put(cc)
    cc.start()

Current Output

hi
hi
hi
donedonedone

Expected Output

hi
done   
hi
done
hi
done

*"I need to run the function multiple times but only after a single run has been completed"* doesn't this defeat the purpose of parallel processing? Looks like you might want a simple loop? — mozway, Dec 27 '21 at 07:14
You would need [mutex lock](https://stackoverflow.com/questions/34524/what-is-a-mutex) check [this](https://stackoverflow.com/a/3311157/5319180) out for an example usage. — D.B.K, Dec 27 '21 at 07:17
thanks mozway yeah i am aware regular for loop wil work on this scenario just wandering how i can use with Queue so later if required i can use it with complex functions — , Dec 27 '21 at 07:18

Yevhen Bondar · Accepted Answer · 2021-12-27T07:53:08.820

1

You can use a Semaphore for your purposes

A semaphore manages an internal counter which is decremented by each acquire() call and incremented by each release() call. The counter can never go below zero; when acquire() finds that it is zero, it blocks, waiting until some other thread calls release().

A default value of Semaphore is 1,

class threading.Semaphore(value=1)

so only one thread would be active at once:

import queue
import threading
import time

fifo_queue = queue.Queue()

semaphore = threading.Semaphore()


def hd():
    with semaphore:
        print("hi")
        time.sleep(1)
        print("done")


for i in range(3):
    cc = threading.Thread(target=hd)
    fifo_queue.put(cc)
    cc.start()

hi
done
hi
done
hi
done

As @user2357112supportsMonica mentioned in comments RLock would be more safe option

class threading.RLock

This class implements reentrant lock objects. A reentrant lock must be released by the thread that acquired it. Once a thread has acquired a reentrant lock, the same thread may acquire it again without blocking; the thread must release it once for each time it has acquired it.

import queue
import threading
import time

fifo_queue = queue.Queue()

lock = threading.RLock()


def hd():
    with lock:
        print("hi")
        time.sleep(1)
        print("done")


for i in range(3):
    cc = threading.Thread(target=hd)
    fifo_queue.put(cc)
    cc.start()

edited Dec 27 '21 at 07:53

answered Dec 27 '21 at 07:18

Yevhen Bondar

4,357
1
11
31

Might be worth pointing out that the default value of the semaphore is 1 ;) – mozway Dec 27 '21 at 07:24
@mozway good point, thx. Added to answer – Yevhen Bondar Dec 27 '21 at 07:27
2

If you're just going to use a semaphore like this, a lock would be more appropriate, either `threading.Lock` or `threading.RLock`. (`RLock` is usually the safest default, avoiding several classes of bugs that can happen with `Lock` or `Semaphore`.) – user2357112 Dec 27 '21 at 07:30
@user2357112supportsMonica Yes, `RLock` seems better for this task. Could you tell about bugs that can happen with `Semaphore`? – Yevhen Bondar Dec 27 '21 at 07:36
1

@Eugenij: If a thread tries to release an `RLock` a different thread is holding, you get a `RuntimeError`, letting you know about the problem. If you try that with a `Lock` or `Semaphore`, the release silently succeeds, hiding bugs. – user2357112 Dec 27 '21 at 07:44
1

Plus, if a thread tries to acquire a lock it's already holding, with a `Lock` or 1-count `Semaphore`, it silently deadlocks. With an `RLock`, the second acquire succeeds, and the lock is only unlocked once the thread unlocks it twice. This is important for cases where you have functions that do `with lock: ...` delegating to other functions that also need to do `with lock: ...`. – user2357112 Dec 27 '21 at 07:50
@Eugenij but how you release the thread from the queue after it's done? Should you put in the end of `hd()` function `fifo_queue.get()` command? will it be danger a bit? – FreeGor May 18 '22 at 06:33

score 0 · Answer 2 · answered Dec 27 '21 at 07:36

please put the print("down") before sleep. it will work fine. Reason: your program will do this: thread1
: print
sleep
print but while the thread is sleeping, other threads will be working and printing their first command. in my way the thread will write the first, write the second and then go to sleep and wait for other threads to show up.

How to use Queue with threading properly

2 Answers2