tqdm progressbar and zip built-in do not work together

Question

tqdm is a Python module to easily print in the console a dynamically updating progressbar. For example

from tqdm import tqdm
from time import sleep
for _ in tqdm(range(10)): 
    sleep(0.1)

prints a dynamic progressbar in the console for 1sec as the iteration executes:

I have not figured out how to use tqdm with the builtin zip object.
The use case of this would be to iterate over two corresponding lists with a console progressbar.
For example, I would expect this to work:

for _, _ in tqdm(zip(range(10), range(10))):
    sleep(0.1)

but the progressbar printed to the console in this case is not correct:

A workaround is to use tqdm with enumerate, however then an iterator index must be defined and managed.

score 92 · Accepted Answer · edited Feb 09 '21 at 10:10

92

tqdm can be used with zip if a total keyword argument is provided in the tqdm call.

The following example demonstrates iteration over corresponding elements in two lists with a working __tqdm__ progress bar for the case where a total keyword argument is used:

The issue is that tqdm needs to know the length of the iterable ahead of time. Because zip is meant to handle iterables with different lengths, it does not have as an attribute a single length of its arguments.

So, __tqdm__ still works nicely with zip, you just need to provide a little manual control with the total keyword argument.

edited Feb 09 '21 at 10:10

Ivan

34,531
8
55
100

answered Dec 16 '16 at 00:28

Russell Burdt

2,391
2
19
30

1

To further automate this, you can go ahead and have a list like `list = [numbers, letters]` and get the largest list by `longest = list[sorted([(i,len(l)) for i,l in enumerate(list)], key=lambda t: t[1])[-1][0]]` then simply put `total=len(longest)`. This approach will also work for `itertools.zip_longest`. – Muneeb Ahmad Khurram Mar 10 '23 at 14:40

casper.dcl · Answer 2 · 2021-01-07T16:20:53.847

Using tqdm>=4.42.0, you should do:

from tqdm.contrib import tzip
from time import sleep

for _, _ in tzip(range(10), range(10)):
    sleep(0.1)

Just to note in https://github.com/tqdm/tqdm#faq-and-known-issues:

Wrapping generators:

Generator wrapper functions tend to hide the length of iterables. tqdm does not.

Replace tqdm(enumerate(...)) with enumerate(tqdm(...)) or tqdm(enumerate(x), total=len(x), ...). The same applies to numpy.ndenumerate.

Replace tqdm(zip(a, b)) with zip(tqdm(a), b) or even zip(tqdm(a), tqdm(b)).

The same applies to itertools.

Some useful convenience functions can be found under tqdm.contrib.

score 5 · Answer 3 · edited May 23 '17 at 11:46

For you have a progress-bar is expected that you can predict the length of your data structure.

range implements the hook method __len__, so you can discover the length doing built-in len

>>> dir(range(10))
[ '__le__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__len__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'count', 'index', 'start', 'step', 'stop']

>>> len(range(10))
10

zip, however, does not provide a way to guess the length of the wrapped structure, so probably that's why because tqdm can not show the progress bar.

dir(zip(range(10))) # no __len__ here
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__lt__', '__ne__', '__new__', '__next__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']

>>> len(zip(range(10)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object of type 'zip' has no len()

Edit:

Yeah, that's it. Look the docs

...

Wrapping enumerated iterables: use enumerate(tqdm(...)) instead of tqdm(enumerate(...)). The same applies to numpy.ndenumerate. This is because enumerate functions tend to hide the length of iterables. tqdm does not.

...

Manual control on tqdm() updates by using a with statement:
with tqdm(total=100) as pbar:
    for i in range(10):
        pbar.update(10)
If the optional variable total (or an iterable with len()) is provided, predictive stats are displayed.

with is also optional (you can just assign tqdm() to a variable, but in this case don't forget to del or close() at the end:
pbar = tqdm(total=100)
for i in range(10):
    pbar.update(10)
pbar.close()

Thanks for this post. It helped figure out a way to initialize tqdm with the 'total' keyword argument that makes it work with zip. — Russell Burdt, Dec 16 '16 at 00:29

score -1 · Answer 4 · answered Sep 17 '22 at 18:00

-1

from tqdm import tqdm

a = range(10)
b = range(10)

for _ in zip(tqdm(a), b):
    pass

answered Sep 17 '22 at 18:00

jitter

346
5
12

4

Code is a lot more helpful when it is accompanied by an explanation. Stack Overflow is about learning, not providing snippets to blindly copy and paste. This is particularly important when answering old questions with existing answers (this question is nearly six years old). Please [edit] your answer and explain how it answers the specific question being asked, and how it improves upon what is already here. See [answer]. – ChrisGPT was on strike Sep 19 '22 at 01:51

tqdm progressbar and zip built-in do not work together

4 Answers4