How to make a generator callable?

Question

I'm trying to create a dataset from a CSV file with 784-bit long rows. Here's my code:

import tensorflow as tf

f = open("test.csv", "r")
csvreader = csv.reader(f)
gen = (row for row in csvreader)
ds = tf.data.Dataset()
ds.from_generator(gen, [tf.uint8]*28**2)

I get the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-22-4b244ea66c1d> in <module>()
     12 gen = (row for row in csvreader_pat_trn)
     13 ds = tf.data.Dataset()
---> 14 ds.from_generator(gen, [tf.uint8]*28**2)

~/Documents/Programming/ANN/labs/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py in from_generator(generator, output_types, output_shapes)
    317     """
    318     if not callable(generator):
--> 319       raise TypeError("`generator` must be callable.")
    320     if output_shapes is None:
    321       output_shapes = nest.map_structure(

TypeError: `generator` must be callable.

The docs said that I should have a generator passed to from_generator(), so that's what I did, gen is a generator. But now it's complaining that my generator isn't callable. How can I make the generator callable so I can get this to work?

EDIT: I'd like to add that I'm using python 3.6.4. Is this the reason for the error?

Answer 1

The generator argument (perhaps confusingly) should not actually be a generator, but a callable returning an iterable (for example, a generator function). Probably the easiest option here is to use a lambda. Also, a couple of errors: 1) tf.data.Dataset.from_generator is meant to be called as a class factory method, not from an instance 2) the function (like a few other in TensorFlow) is weirdly picky about parameters, and it wants you to give the sequence of dtypes and each data row as tuples (instead of the lists returned by the CSV reader), you can use for example map for that:

import csv
import tensorflow as tf

with open("test.csv", "r") as f:
    csvreader = csv.reader(f)
    ds = tf.data.Dataset.from_generator(lambda: map(tuple, csvreader),
                                        (tf.uint8,) * (28 ** 2))

Answer 2

Yuck, two years later... But hey! Another solution! :D

This might not be the cleanest answer but for generators that are more complicated, you can use a decorator. I made a generator that yields two dictionaries, for example:

>>> train,val = dataloader("path/to/dataset")
>>> x,y = next(train)
>>> print(x)
{"data": [...], "filename": "image.png"}

>>> print(y)
{"category": "Dog", "category_id": 1, "background": "park"}

When I tried using the from_generator, it gave me the error:

>>> ds_tf = tf.data.Dataset.from_generator(
    iter(mm),
    ({"data":tf.float32, "filename":tf.string},
    {"category":tf.string, "category_id":tf.int32, "background":tf.string})
    )
TypeError: `generator` must be callable.

But then I wrote a decorating function

>>> def make_gen_callable(_gen):
        def gen():
            for x,y in _gen:
                 yield x,y
        return gen
>>> train_ = make_gen_callable(train)

>>> train_ds = tf.data.Dataset.from_generator(
    train_,
    ({"data":tf.float32, "filename":tf.string},
    {"category":tf.string, "category_id":tf.int32, "background":tf.string})
    )

>>> for x,y in train_ds:
        break

>>> print(x)
{'data': <tf.Tensor: shape=(320, 480), dtype=float32, ... >,
 'filename': <tf.Tensor: shape=(), dtype=string, ...> 
}

>>> print(y)
{'category': <tf.Tensor: shape=(), dtype=string, numpy=b'Dog'>,
 'category_id': <tf.Tensor: shape=(), dtype=int32, numpy=1>,
 'background': <tf.Tensor: shape=(), dtype=string, numpy=b'Living Room'>
}

But now, note that in order to iterate train_, one has to call it

>>> for x,y in train_():
        do_stuff(x,y)
        ...

Answer 3

From the docs, which you linked:

The generator argument must be a callable object that returns an object that support the iter() protocol (e.g. a generator function)

This means you should be able to do something like this:

import tensorflow as tf
import csv

with open("test.csv", "r") as f:
    csvreader = csv.reader(f)
    gen = lambda: (row for row in csvreader)
    ds = tf.data.Dataset()
    ds.from_generator(gen, [tf.uint8]*28**2)

In other words, the function you pass must produce a generator when called. This is easy to achieve when making it an anonymous function (a lambda).

Alternatively try this, which is closer to how it is done in the docs:

import tensorflow as tf
import csv


def read_csv(file_name="test.csv"):
    with open(file_name) as f:
        reader = csv.reader(f)
        for row in reader:
            yield row

ds = tf.data.Dataset.from_generator(read_csv, [tf.uint8]*28**2)

(If you need a different file name than whatever default you set, you can use functools.partial(read_csv, file_name="whatever.csv").)

The difference is that the read_csv function returns the generator object when called, whereas what you constructed is already the generator object and equivalent to doing:

gen = read_csv()
ds = tf.data.Dataset.from_generator(gen, [tf.uint8]*28**2)  # does not work

Answer 4

Pushing 5 years late, but given the two-hour-long headache this has caused me -- for those who want a more generalisable solution:

class TFDatasetFromGenerator():

    def __init__(self, somearg=""):
        self.someparam = somearg
        '''etc; eg load S3 keys into a dataframe, s3keyDF'''
        self.DataFrame= s3keyDF

    def some_modifying_function(self, someotherarg=""):
        ''' do some stuff, 
        eg on self.someparam etc 
        or split self.DataFrame into train/test data refs etc
        '''
 
    def generate(self, arg1="", arg2=True, batch_size=128):
        return tf.data.Dataset.from_generator(
            self._generate(
                arg1=arg1,
                arg2=arg2,
                batch_size = batch_size
             ),
             (tf.float32, tf.float32),
             ((None,128,400,4), (None,1))
         )    
         ''' modify the data output shape as required; 
         we present x and y (input data and labels) as a tuple
         with both x and y being tf.float32
         x having the shape (batch_size,128,400,4)
         y having the shape (batch_size,1).
         '''

    def _generate(self, arg1="", arg2=True, batch_size=128):
        ''' _generate defines and returns the function `gen`.
        returning the function makes _generate a callable
        _generate can therefore be used as the first argument 
        to tf.data.Dataset.from_generator()
        '''

        def gen():

            '''do preprocessing, eg filter self.dataset 
            to return just training or test portions of the dataset    
            eg:
            '''
            if(arg1="train"):
                data_refs = list(
                    self.TrainDataFrame.data_referance.unique()
                )
                label_refs = list(
                    self.TrainDataFrame.label_refernce.unique()
                )
            ''' etc '''
            random.shuffle(data)
            

            ## get the sample count
            sample_count  = len(data_refs)

            while True:
                sample_index = 0

                while sample_index < sample_count:
                    start        = sample_index
                    end          = min(start + batch_size, sample_count)

                    # Load data from DataFames
                    x = data_refs[start: end]
                    y = label_refs[start: end]

                data = []
                for key in x:
                    ''' d = load your data '''
                    data.append(d)

                labels = []
                for key in y:
                    ''' l = load your labels '''
                    labels.append(l)
                

                sample_index += batch_size

                yield (np.array(data),np.array(labels))

        return gen

Hope this helps someone!