# Why is using keras ImageDataGenerator for data augmentation relevent?

I have used keras ImageDataGenerator to generate more data in my neural networks as I have had really small datasets and it has proven itself. As far as I understand, it generates new batches each epoch on the fly and so the neural network is trained on a "little bit" new data on each iteration.

Why/how is this method relevant? Is there any research done on this topic? I just haven't seen it in my studies so far and want to know whether this is commonly used method or just some feature that keras offers?

• This methodology is not applicable to CNNs, or even neural networks. For example, consider the bootstrap. All ImageDataGenerator is doing is sampling from the empirical CDF and adding some noise. This should all approximate the theoretical CDF that the neural network is trying to learn. – Jon May 4 at 18:09

Yes, there's sophisticated literature base on this topic. At this point, I think it's safe to say that that transforming image inputs is widely accepted as a way to improve the robustness of a network, so the practical question becomes which transformations are best, and how to cheaply go about generating good transformations for particular use-cases.

The basic idea is that if you're doing an image classification task, the network should do a good job of detecting the same object even if it's positioned differently within the frame (translated, rotated, etc). In the most general setting, the purpose is to simulate the fact that there's no particular need for the photographer to have taken a picture with a specific composition or arrangement of the objects. The same reasoning applies to varying brightness and other image attributes.

Intuitively: A dog is a dog, whether the photo is taken head-on, from above, or from the side, or if the dog is illuminated by dim or bright light.

What augmentations are "admissible" depend on your goals. Not all augmentations align with specific tasks, so training on irrelevant modifications are unlikely to improve your classifier. For example, if you translate an image too much and the target object falls out of frame, then there's no longer relevant semantic content to classify.

Some papers discussing transformations of images to improve neural networks:

• "Deep Convolutional Inverse Graphics Network" Tejas D. Kulkarni, Will Whitney, Pushmeet Kohli, Joshua B. Tenenbaum includes transformations of images as a step.

• Perusing the top results for CIFAR-10 reveals that data augmentation is often a step in the pipeline.

• Augmentation can even be unsupervised, which can be important if labels are scarce or expensive. Example: "Unsupervised Data Augmentation" by Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, Quoc V. Le.

• I know why data augmentation is used, but ImageDataGenerator does it somewhat differently. Instead of generating new images and making the dataset larger and then training on that dataset, it uses a little different batches every time. Can CNN really learn well if it sees different data in every iteration? – Andry May 4 at 18:53
• If your only goal is to predict well against the training data, then simulating new data "online" might get in the way of perfectly predicting the training data. On the other hand, the more relevant task in nearly every context is to predict well on out-of-sample data; sampling from the desired distribution over inputs seems like a plausible way to go about achieving the task. While the location of a true minimum might be different for each sample, I'd happily trade that for better generalization since guarding against overfitting is usually the larger part of the battle. – Sycorax May 4 at 19:12
• The question "Can CNN really learn well if it sees different data in every iteration?" can also be viewed as an empirical question, for which the resources in my answer provide the answer "Yes!" – Sycorax May 4 at 19:16