Custom generator object for lazily flattening large datasource

Question

I'm looking for the object oriented equivalent of this function:

def lazy_gen_func(path):
    for line in open(path): 
        for token in line.split():
            yield token

Related answers suggest the following approach:

class eager_gen_obj(object):
    def __init__(self, path):
        f = open(path)
        self.text = [token for line in f for token in line.split()]
        self.index = 0

    def __iter__(self):
        return self

    def __next__(self):
        try:
            result = self.text[self.index]
        except IndexError:
            raise StopIteration
        self.index += 1
        return result

With the downside that the full source file has to be loaded in memory when __init__ is called.

How do I create a custom generator object to lazily flatten nested source data?

Re: your suggested edit: you can’t use `with` when creating the generator (it’ll close the file immediately), hence the suggestion at the end to not leave `f` up to GC. (The remaining changes were also intentional.) — Ry-, Feb 20 '18 at 16:29
see also my second suggested edit to address this. I tested the with statement and it did not misbehave. — Tom Rijntjes, Feb 20 '18 at 16:40
Ah, you probably tested it with `readlines`. `readlines` is not lazy. (Suggested edits get merged, by the way – I only saw the last version.) — Ry-, Feb 20 '18 at 16:51

score 1 · Accepted Answer · answered Feb 20 '18 at 16:18

You can wrap a generator, either from the original function – self._generator = lazy_gen_func(path) – or from an equivalent generator expression:

class LazyGenObj:
    def __init__(self, path):
        f = open(path)
        self._generator = (token for line in f for token in line.split())

    def __iter__(self):
        return self

    def __next__(self):
        return next(self._generator)

It might also be a good idea to pass the file object to this instead of a path so the caller is free to close it explicitly.

Custom generator object for lazily flattening large datasource

1 Answers1