Frequency analysis of values produced by generator

Question

I'm doing some analysis of images, and I have a generator that gives me all pixels in image:

def pixels_g(img):
    w, h = img.shape
    for y in range(0, h):
        for x in range(0, w):
            yield img[y][x]

It's output if converted to list would be something like

[0, 1, 2, 5, 240, 5, ... ]

Now I'd like to get "frequency table" for it:

{
    0: 0,
    1: 5,
    2: 10,
    3: 0,
    4: 0,
    #snip
    255: 7
}

I've found some useful options in this question, but they all work with list, and I don't think that creating a list from my generator is a good idea - it can have millions of elements.

I'm therefore looking for a way to do this while preserving the benefits of generator. I'll process many images and don't want to hog resources too much.

collections.counter strikes yet again. This question definitely has at least a dozen duplicates. — Oleh Prypin, Dec 09 '14 at 22:49
@OlehPrypin will collections.Counter take advantage of it being a generator? — MightyPork, Dec 09 '14 at 22:50
The only difference may be that it doesn't save zero-count elements... — Oleh Prypin, Dec 09 '14 at 22:51

pbkhrv · Answer 1 · 2014-12-09T23:00:43.580

0

Use Counter from "collections". It works with any iterable, not just lists.

from collections import Counter

pixels = pixels_g(img)
c = Counter(pixels)

print c[4]

If you need to reuse the contents of 'pixels' generator instance after running it through Counter, use 'tee' from itertools:

from collections import Counter
from itertools import tee

(pixels, hist) = tee(pixels_g(img))
c = Counter(pixels)
# can use hist for something else

edited Dec 09 '14 at 23:00

answered Dec 09 '14 at 22:52

pbkhrv

647
4
11

that would consume the generator – Padraic Cunningham Dec 09 '14 at 22:53
Works, except I don't get zeros for missing values. `dict.get(num, 0)` might deal with it, I guess... but it's a hack – MightyPork Dec 09 '14 at 22:53
@PadraicCunningham what do you mean, consume? – MightyPork Dec 09 '14 at 22:54
I mean if you called counter on it and tried using the generator after you would get nothing – Padraic Cunningham Dec 09 '14 at 22:55
@MightyPork A Counter object already returns 0 for missing items, so `.get()` is not required. – Ashwini Chaudhary Dec 09 '14 at 22:56
Not if I convert it to a dict. I intend to plot it as histogram, there it might be a problem. – MightyPork Dec 09 '14 at 22:58
if you need a dict() that explicitly contains 0 counts, you could use a defaultdict and init it with something like `zip(range(0,256), itertools.repeat(0, times=256))`, maybe – pbkhrv Dec 09 '14 at 23:16

Frequency analysis of values produced by generator

1 Answers1