5
$\begingroup$

Data augmentation is popularly done online as that is how it is typically implemented and suggested in neural network frameworks like Keras and TensorFlow. I have also seen it described in e.g. the AlexNet paper.

Online data augmentation implies that the network will see a completely different dataset every epoch. Superficially this seems like a cool idea (especially because it is computationally free if you augment the next batch on the CPU and train the previous batch on the GPU) with some people saying that it improves generalization, but other than that I did not find quality literature supporting and explaining this idea.

My issue with online data augmentation is that I thought gradient-based learning algorithms were fundamentally based on repetition (i.e. seeing the same dataset every epoch) which would make sense intuitively. Is this actually the case though or did I make it up? Is there any literature covering this?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

From an optimization standpoint, repetition is nice (we want to optimize the same function). From a modeling standpoint, repetition can risk memorizing the training data without learning anything generalizable. For image data, online augmentation is motivated by observing that we can translate or add noise or otherwise distort an image but retain its key semantic content (so a human can still recognize it). The hypothesis of online augmentation is that the model probably won't see the exact same image twice, so memorization is unlikely, so the model will generalize well.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Could you please elaborate on why repetition is nice from an optimization standpoint? Or maybe point me to optimization literature that explains this? $\endgroup$
    – fabiomaia
    Commented Mar 25, 2019 at 19:05
  • 2
    $\begingroup$ It's pretty straightforward. Suppose you want to minimize $\| x^\top \theta - y\|_2^2$ for some data $x$ and target $y$ by incrementally updating $\theta$ and you're using gradient descent. If you augment your data online, you're changing $x$ at each iteration, and therefore you're changing the surface and the location of the minimizer $\theta^*$. $\endgroup$
    – Sycorax ♦
    Commented Mar 25, 2019 at 19:07
  • $\begingroup$ There's also a connection to wider minima generalizing better and augmentation in a sense regularizes / prevents a neural network from settling down into a very narrow minimum that does not generalize, right? $\endgroup$
    – Björn
    Commented May 4, 2021 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.