Python Threading Queue vs multiprocessing Pipe

Question

I am building a python application that utilizes Pyqt4/Pyqtgraph as the GUI framework and a BLE module. The current application receives data from the BLE module and displays it in real time.

I wrote some benchmark code to evaluate the overhead relating to moving the BLE module into a separate process. However, the results turned out to be confusing.

Streaming data utilizing Pipes between processors are faster than streaming data between threads utilizing Queues. I always assumed communicating within a processor is faster.

What is happening?

Thanks in advance.

These are the codes I used

Two processes communicating with Pipes

from  multiprocessing import Process, Pipe

def worker(q):
    output_p, input_p = q
    input_p.close()
    for task_nbr in range(100000):
        message = output_p.recv()
    sys.exit(1)

def main():
    output_p, input_p = Pipe()
    Process(target=worker, args=((output_p,input_p),)).start()
    b = [i for i in range(20)]
    start_time = time.time()
    for num in range(100000):
        input_p.send(b)
    end_time = time.time()

    duration = end_time - start_time
    msg_per_sec = 100000 / duration

    print "Duration: %s" % duration
    print "Messages Per Second: %s" % msg_per_sec

if __name__ == "__main__":
    main()

Two threads communicating with Queue

from Queue import Queue
from threading import Thread
def worker(q):
    for x in range(100000):
        q.get("MESSAGE")
def main():
    q = Queue()
    t = Thread(target=worker,args=(q,))
    t.start()

    start_time = time.time()
    b = [i for i in range(20)]
    print b
    for x in range(100000):
        q.put(b)
    end_time = time.time()

    duration = end_time - start_time
    msg_per_sec = 100000 / duration

    print "Duration: %s" % duration
    print "Messages Per Second: %s" % msg_per_sec

if __name__ == "__main__":
    main()

Answer 1

This stack overflow question give a well performance comparison between Pipe() and Queue().

With multiprocessing, two processes can run parallel without caring for the global interpreter lock (GIL) as Developer Richtofen mentioned, while multi-threading spawns two threads in the same process space. So multiprocess with a careful design performs better than multi-threading.

Answer 2

There is sth Called GIL (Global Interpreter Lock) which doesn't let python properly work with multiple Threads on CPU-Bound Computing (just like this in case) Also you should consider use Threads at I/O-Bound Computing (like waiting for User-/File or Function Input) and Process for CPU-Bound Computing (like calculate multiples algorithms or in this case For Loops)