multiprocessing.Pipe() vs .Queue()

Question

import multiprocess as mp

What are the key differences between mp.Pipe() and mp.Queue()? They seem to be the same to me: basically Pipe.recv() is equivalent to Queue.get(), and Pipe.send() is Queue.put().

Answer 1

They're very different things, with very different behavior.

A Queue instance has put, get, empty, full, and various other methods. It has an optional maximum size (number of items, really). Anyone can put or get into any queue. It is process-safe as it handles all the locking.

The Pipe function—note that this is a function, not a class instance—returns two objects of type Connection (these are the class instances). These two instances are connected to each other. The connection between them can be single-duplex, i.e., you can only send on one and only receive on the other, or it can be full-duplex, i.e., whatever you actually send on one is received on the other. The two objects have send, recv, send_bytes, recv_bytes, fileno, and close methods among others. The send and receive methods use the pickling code to translate between objects and bytes since the actual data transfer is via byte-stream. The connection objects are not locked and therefore not process-safe.

Data transfer between processes generally uses these Connection objects: this and shared-memory are the underlying mechanism for all process-to-process communication in the multiprocessing code. Queue instances are much higher level objects, which ultimately need to use a Connection to send or receive the byte-stream that represents the object being transferred across the queue. So in that sense, they do the same thing—but that's a bit like saying that a USB cable does the same thing as the things that connect them. Usually, you don't want to deal with individual voltages on a wire: it's much nicer to just send or receve a whole object. (This analogy is a little weak, because Connection instances have send and recv as well as send_bytes and recv_bytes, but it's probably still helpful.)