I'm training a convolutional neural network by adding small perturbations (like rotation and shifting) to the images each time it gets a batch data.

I think a better way of doing this could be to generate a bigger data set by oversampling with these transformations before training, however I don't have much RAM to store a bigger data set, so I decided to transform them on the run. But it means that now I'm supplying a different data set in each epoch.

I think it should make sense, and the training, in fact, converges to a better performance, but it watches a zig-zaggy road, which is understandable I guess. So am I safe to assume that I'm not making a mistake? Also is there a better way for this, i.e. training a generalizable classifier under a small data set and limited computational resources?

Thanks for any help !

  • $\begingroup$ I don't see anything wrong with doing that, it's actually a nice solution to a hardware constraint problem. Dropout, a very popular regularization technique for deep nets, also introduces noise during each forward pass, so in my opinion there is nothing wrong with these sorts of techniques. Nicely done! $\endgroup$
    – Indie AI
    Commented Dec 30, 2015 at 21:45

1 Answer 1


Yes, it does make sense and actually this is what is done in practice.

Why does it makes sense?

Random permutations might change the statistics of the dataset. By performing a new random permutation in each epoch, these effects tend to cancel out.

Imagine performing a random rotation on an image with a range of $[-5\%, +5\%]$. By storing the rotated image, you are feeding the same image to the network during each epoch. By transforming the image anew during each epoch, a different image is fed to the network.

This both increases the effective size of your augmented dataset and reduces the bias caused by the augmentation (because you don't perform the same perturbation each time).

Note on the memory issues:

When dealing with images, it is preferable to not load the whole dataset in your memory during training; in most cases datasets are too large to fit into your memory. I'd recommend using a generator to load image batches one at a time. This way you don't have to concern yourself on how large the dataset is.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.