Why is using deque of Queue.Queue considered dangerous?

Question

In a program where I store some dict in a Queue.Queue and I have to list each entry without removing them from the queue. Now I use this code snippet to iterate over the entries:

elements = Queue.Queue()
# populate the queue

for elem in list(elements.queue):
    # print elem

In the code of Lib/Queue.py and Lib/collections.py I found that queue is a proxy to deque and in this answer of "Python: Queue.Queue vs. collections.deque" is mentioned not to use deque directly.

Is there any particular reason not to use deque? Hope someone can elaborate a little bit.

I think you're misunderstanding the answer. There's nothing wrong with using `collections.deque`, you just don't want to use `Queue.deque` when using a `Queue` object. That is the internal deque of the `Queue` object, and you shouldn't directly manipulate it. — Roger Fan, Oct 03 '14 at 22:43
The explanation is fairly clear in the question you linked to... If you just need a data structure then a deque is fine. — beoliver, Oct 03 '14 at 22:43
https://docs.python.org/2/library/collections.html#collections.deque — beoliver, Oct 03 '14 at 22:46

dano · Accepted Answer · 2014-10-03T23:00:35.497

The reason its dangerous to iterate over the internal deque of a Queue.Queue is that another thread could come along and add or remove items from the deque while you're iterating over it (by using queue.put/queue.get). This means that your iterable could change size during iteration, which causes undefined behavior in Python, and is not something you want to happen. You can iterate over the deque safely, though, by acquiring the internal mutex of the Queue first:

elements = Queue.Queue()
# populate the queue

with elements.mutex:
    for elem in list(elements.queue):
        # print elem

This way, all other threads will be prevented from adding or removing elements from the Queue while you iterate.

Now, this has nothing to do with using deque in general - which is a perfectly valid thing to do. It's just within the context of using the deque that's internal to a Queue.Queue that you need to be careful.

Why is using deque of Queue.Queue considered dangerous?

1 Answers1