Keras ImageDataGenerator [closed]

Question

I find the documentation and tutorials on the Internet surrounding ImageDataGenerator (the data augmentation function for Keras) to not really explain much at all how it works.

Following the instructions here also leads room for a lot of interpretation: https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

I have two questions. First, am I supposed to run model.fit_generator in addition to the normal model.fit? If so, how do I do this? When I try running model.fit_generator and then model.fit I find my validation loss increases (and accuracy decreases) when running model.fit after model.fit_generator. If I'm not supposed to use model.fit at all, wouldn't this be ignoring the non-augmented training set data?

Secondly, I don't understand the example in the above blog post where we input a validation generator for the model.fit_generator instead of using non-augmented data for the validation set. The entire point of the validation set is to see how the model predicts real world data, not how well it can predict synthetic augmentation data that is of no use. So then, what is the point of augmenting the validation set? How well the model predicts augmented data tells me nothing about how well it can predict real-world data, and I thought this augmented data would only be useful for the training set, not the validation set.

Thanks.

Answer 1

First, am I supposed to run model.fit_generator in addition to the normal model.fit?

Just model.fit_generator will do the job. Calling model.fit_generator then model.fit will do the training twice with and without augmentation.

If I'm not supposed to use model.fit at all, wouldn't this be ignoring the non-augmented training set data?

Right, except that there's a (small) probability the generator will just give you the non-augmented data by chance. So if you want to mix augmented and non-augmented data, you might need some extra work.

So then, what is the point of augmenting the validation set?

There're many reasons for doing test time augmentation, for example if you train a network for detecting faces in images at some certain scale, and at test time you want to deal with images of different resolutions, you can augment the images by different scales to be able to detect faces at all scales.

An other example provided in this paper shows that test time augmentation can lead to better performance on image recognition tasks.

Answer 2

Alright, TL;DR:

1. You shouldn't call model.fit after model.fit_generator
2. No, validation data is not the measure on real world data.

More in depth answer:

1. The generator you create IS the data augmentation, if you do model.fit you probably have non-augmented data as X. This will probably overfit very quickly.
2. Data is usually split into two chunks training-data and test-data. Afterwards a validation-data is chosen as a subset of training-data. Sometimes validation-data is also non-intersecting with training-data. The metric calculation on huge datasets is costly and using only a fraction (the validation-data) as the target, you get faster training, because less time is spent trying to aggregate feedback metric (such as precision). What you probably want, is to have training and validation generators constructed after train/test data split and at the end of training get test precision.

Answer 3

Call fit_generator doesn't mean you augment the data. In the test_generator the only thing it does is rescale the image. So no augmentation is done on validation set.