5
$\begingroup$

In deep learning papers, data augmentation is often presented as a type of regularization. For example, this is explored in Chiyan Zhang and coauthor's presentation at ICLR17, Understanding deep learning requires rethinking generalization. Why is this classification given? Intuitively, I see data augmentation as a way of expanding a dataset, but regularization as a means of modifying a training algorithm to (hopefully) improve the generalization error.

$\endgroup$
  • 2
    $\begingroup$ Could you point us to an article/link stating this? $\endgroup$ – Pop Jul 31 '17 at 12:27
9
$\begingroup$

Regularization (traditionally in the context of shrinkage) adds prior knowledge to a model; a prior, literally, is specified for the parameters. Augmentation is also a form of adding prior knowledge to a model; e.g. images are rotated, which you know does not change the class label. Increasing training data (as with augmentation) decreases a model's variance. Regularization also decreases a model's variance. They do so in different ways, but ultimately both decrease regularization error.

Section 5.2.2 of Goodfellow et al's Deep Learning proposes a much broader definition:

Regularization is any modification we make to a learning algorithm that is intended to reduce its generalization error but not its training error.

There is a tendency to asssociate regularization with shrinkage because of the term "l-p norm regularization"...perhaps "augmentation regularization" is equally valid, although it doesn't roll off the tongue.

$\endgroup$
  • $\begingroup$ Thanks, this is actually a good intuitive explanation. With regards to your citation, my main misunderstanding comes from: "... modifications to a learning algorithm ...". I suppose I see data augmentation as a preprocessing step, separate to the learning algorithm $\endgroup$ – Gilly Jul 31 '17 at 23:56
  • 1
    $\begingroup$ Yes, agreed. Augmentation really is a way to modify the model, however, to reduce variance, not a step to transform the data such that it is appropriate for learning, which is "preprocessing." $\endgroup$ – user0 Aug 1 '17 at 0:02
5
$\begingroup$

User99889 had a good answer (+1), I will add some comments to elaborate the points.

The goal of regularization is reducing the Variance and increase Bias in Bias Variance Trade Off. This can be achieved in different ways, here are some examples:

  • Increase the amount of the data, collect new data or derive new data from existing data / data augmentation.
  • Reduce the complex of the model. For example, reduce the hidden layers or number of units in the hidden layer.
  • Put constraints in the coefficients/parameters. For example, L1 or L2 regularization.

We will further to explain why User99889 mentioned about "add prior / domain knowledge". Let use MNIST data as an example, we can create more data by using the following knowledge:

  • If we re-scale digit, the label will not be changed
  • If we shift digit, the label will not be changed

Then, we have more data using domain knowledge and the model will have more Bias.

$\endgroup$
  • $\begingroup$ So does this mean, also, that providing more data is a form of regularization? By extension, is using better quality data a form of regularization? $\endgroup$ – Gilly Jul 31 '17 at 23:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.