6

I am aware of multiprocessing.Manager() and how it can be used to create shared objects. In particular, queues which can be shared among workers. There is this question, this question, and this question.

However, these links don't mention why we can use inheritance for sharing between processes. As I understand, a queue can still only be copied in this case.

Community
  • 1
  • 1
del bao
  • 1,084
  • 1
  • 11
  • 20
  • 4
    Are you thinking of the wrong meaning of "inheritance"? They're talking about processes inheriting queues from the process that spawns them, not `class Child(Parent)` OOP inheritance. – user2357112 Mar 07 '17 at 00:43
  • pls see this https://stackoverflow.com/a/55577761/429476 – Alex Punnen Apr 05 '22 at 12:35

1 Answers1

6

The Queue implementation in python relies on a system pipe to transmit the data from one process to another and some semaphores to protect the read and write on this pipe.

The pipe is handled as an open file in the process and can only be shared with a subprocess at spawning time, because of OS constraints.
The semaphores are also treated as files that should only be shared at spawning time, at least in UNIX based system, for early version of python.

As these 2 sub objects cannot be shared in general, the Queue cannot be pickled and sent to a subprocess once it has been started.

However, for some OS and recent version of python, it is possible to share the Connection and to create sharable Semaphore. Thus, you could in theory create your own Queue that can be shared between processes. But it involves a lot of hacks and might not be very secured.

Thomas Moreau
  • 4,377
  • 1
  • 20
  • 32
  • For future readers, you can find a code example for the above approach to make queues, locks, etc. picklable here https://stackoverflow.com/a/75247561/16310741 – Charchit Agarwal Feb 19 '23 at 15:00