Does it makes sense to perturb images on-the-run while training CNNs? I'm training a convolutional neural network by adding small perturbations (like rotation and shifting) to the images each time it gets a batch data.
I think a better way of doing this could be to generate a bigger data set by oversampling with these transformations before training, however I don't have much RAM to store a bigger data set, so I decided to transform them on the run. But it means that now I'm supplying a different data set in each epoch. 
I think it should make sense, and the training, in fact, converges to a better performance, but it watches a zig-zaggy road, which is understandable I guess. So am I safe to assume that I'm not making a mistake? Also is there a better way for this, i.e. training a generalizable classifier under a small data set and limited computational resources?
Thanks for any help !
 A: Yes, it does make sense and actually this is what is done in practice.
Why does it makes sense?
Random permutations might change the statistics of the dataset. By performing a new random permutation in each epoch, these effects tend to cancel out.
Imagine performing a random rotation on an image with a range of $[-5\%, +5\%]$. By storing the rotated image, you are feeding the same image to the network during each epoch. By transforming the image anew during each epoch, a different image is fed to the network.
This both increases the effective size of your augmented dataset and reduces the bias caused by the augmentation (because you don't perform the same perturbation each time).
Note on the memory issues:
When dealing with images, it is preferable to not load the whole dataset in your memory during training; in most cases datasets are too large to fit into your memory. I'd recommend using a generator to load image batches one at a time. This way you don't have to concern yourself on how large the dataset is.
