Write custom Data Generator for Keras

Question

I have each datapoint stored in a .npy file, with shape=(1024,7,8). I want to load them to a Keras model by a manner similar to ImageDataGenerator, so I wrote and tried different custom generators but none of them work, here is one I adapted from this

def find(dirpath, prefix=None, suffix=None, recursive=True):
    """Function to find recursively all files with specific prefix and suffix in a directory
    Return a list of paths
    """
    l = []
    if not prefix:
        prefix = ''
    if not suffix:
        suffix = ''
    for (folders, subfolders, files) in os.walk(dirpath):
        for filename in [f for f in files if f.startswith(prefix) and f.endswith(suffix)]:
            l.append(os.path.join(folders, filename))
        if not recursive:
            break
    l
    return l

def generate_data(directory, batch_size):
    i = 0
    file_list = find(directory)
    while True:
        array_batch = []
        for b in range(batch_size):
            if i == len(file_list):
                i = 0
                random.shuffle(file_list)
            sample = file_list[i]
            i += 1

            array = np.load(sample)
            array_batch.append(array)

        yield array_batch

I found this lacks of the label, so it won't be fit into the model using fit_generator . How can I add the label into this generator, given that I can store them in a numpy array?

When while loop will stop in `while True:`?? – N.IT Nov 08 '18 at 19:55 — N.IT, Nov 08 '18 at 19:55

Srihari Humbarwadi · Answer 1 · 2019-09-20T09:50:29.630

15

from tensorflow.python.keras.utils import Sequence
import numpy as np   

class Mygenerator(Sequence):
    def __init__(self, x_set, y_set, batch_size):
        self.x, self.y = x_set, y_set
        self.batch_size = batch_size

    def __len__(self):
        return int(np.ceil(len(self.x) / float(self.batch_size)))

    def __getitem__(self, idx):
        batch_x = self.x[idx * self.batch_size:(idx + 1) * self.batch_size]
        batch_y = self.y[idx * self.batch_size:(idx + 1) * self.batch_size]

        # read your data here using the batch lists, batch_x and batch_y
        x = [my_readfunction(filename) for filename in batch_x] 
        y = [my_readfunction(filename) for filename in batch_y]
        return np.array(x), np.array(y)

edited Sep 20 '19 at 09:50

answered Oct 11 '18 at 08:19

Srihari Humbarwadi

2,532
1
10
28

2

you can now call model.fit_generator() using this generator – Srihari Humbarwadi Oct 11 '18 at 08:20
I apologize if my question is dummy since I'm not good at python, but how would I use this as it's class, not a function? Thank you – Thanh Nguyen Oct 12 '18 at 00:34
1

train_gen = mygenerator(x_set, y_set, batch_size) then use this train_gen in your model.fit_generator call – Srihari Humbarwadi Oct 12 '18 at 06:19
Thanks. It's what I was looking for :) Other solutions in the web are complicated and I couldn't use them. – Masoud Maleki May 21 '20 at 08:21
How does the keras function fit_generator know about the __getitem__ function and in turn get the data to train the model. Or __getitem__ is not required and just self.x and self.y suffice ? – Manmeet Singh Aug 17 '20 at 21:49
Hello. could you let me know how to use it in fit_generator()? i have no idea of how to deal with my_readfunction() – 손상우 Oct 29 '20 at 12:36
You need to replace `my_readfunction` with your own function that processes your data – Srihari Humbarwadi Oct 30 '20 at 13:09
1

The suggested import is tensorflow.keras and not tensorflow.python.keras which is intended only for development use. – Aelius Jun 09 '22 at 15:44

Write custom Data Generator for Keras

1 Answers1

Linked