In deep learning papers, data augmentation is often presented as a type of regularization. For example, this is explored in Chiyan Zhang and coauthor's presentation at ICLR17, Understanding deep learning requires rethinking generalization. Why is this classification given? Intuitively, I see data augmentation as a way of expanding a dataset, but regularization as a means of modifying a training algorithm to (hopefully) improve the generalization error.

  • 2
    $\begingroup$ Could you point us to an article/link stating this? $\endgroup$
    – Pop
    Jul 31, 2017 at 12:27

3 Answers 3


Regularization (traditionally in the context of shrinkage) adds prior knowledge to a model; a prior, literally, is specified for the parameters. Augmentation is also a form of adding prior knowledge to a model; e.g. images are rotated, which you know does not change the class label. Increasing training data (as with augmentation) decreases a model's variance. Regularization also decreases a model's variance. They do so in different ways, but ultimately both decrease regularization error.

Section 5.2.2 of Goodfellow et al's Deep Learning proposes a much broader definition:

Regularization is any modification we make to a learning algorithm that is intended to reduce its generalization error but not its training error.

There is a tendency to asssociate regularization with shrinkage because of the term "l-p norm regularization"...perhaps "augmentation regularization" is equally valid, although it doesn't roll off the tongue.

  • $\begingroup$ Thanks, this is actually a good intuitive explanation. With regards to your citation, my main misunderstanding comes from: "... modifications to a learning algorithm ...". I suppose I see data augmentation as a preprocessing step, separate to the learning algorithm $\endgroup$
    – Gilly
    Jul 31, 2017 at 23:56
  • 1
    $\begingroup$ Yes, agreed. Augmentation really is a way to modify the model, however, to reduce variance, not a step to transform the data such that it is appropriate for learning, which is "preprocessing." $\endgroup$ Aug 1, 2017 at 0:02

User99889 had a good answer (+1), I will add some comments to elaborate the points.

The goal of regularization is reducing the Variance and increase Bias in Bias Variance Trade Off. This can be achieved in different ways, here are some examples:

  • Increase the amount of the data, collect new data or derive new data from existing data / data augmentation.
  • Reduce the complex of the model. For example, reduce the hidden layers or number of units in the hidden layer.
  • Put constraints in the coefficients/parameters. For example, L1 or L2 regularization.

We will further to explain why User99889 mentioned about "add prior / domain knowledge". Let use MNIST data as an example, we can create more data by using the following knowledge:

  • If we re-scale digit, the label will not be changed
  • If we shift digit, the label will not be changed

Then, we have more data using domain knowledge and the model will have more Bias.

  • $\begingroup$ So does this mean, also, that providing more data is a form of regularization? By extension, is using better quality data a form of regularization? $\endgroup$
    – Gilly
    Jul 31, 2017 at 23:52

The objective of regularization is to improve the generalization capability of the model (in other words its ability to perform well on unseen data). Regularization can be explicit or implicit. The first case refers to techniques that can constrain the effective capacity of the model in order to reduce overfitting. Examples include adding a penalty term in standard statistical estimators such as OLS or weight decay and dropout in deep learning. In the second case (implicit regularization) instead of explicitly constraining the capacity of the model we use indirect methods. One example, is using SGD to train linear models as it always converges to a solution with a small norm (https://arxiv.org/pdf/1611.03530.pdf). Similarly, data augmentation (which has many forms e.g. adding predictors, adding artificially created data, resampling) will help the model to generalize better and is thus considered to be an implicit regularization method.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.