Why is data augmentation classified as a type of regularization?

Question

In deep learning papers, data augmentation is often presented as a type of regularization. For example, this is explored in Chiyan Zhang and coauthor's presentation at ICLR17, Understanding deep learning requires rethinking generalization. Why is this classification given? Intuitively, I see data augmentation as a way of expanding a dataset, but regularization as a means of modifying a training algorithm to (hopefully) improve the generalization error.

Answer 1

Regularization (traditionally in the context of shrinkage) adds prior knowledge to a model; a prior, literally, is specified for the parameters. Augmentation is also a form of adding prior knowledge to a model; e.g. images are rotated, which you know does not change the class label. Increasing training data (as with augmentation) decreases a model's variance. Regularization also decreases a model's variance. They do so in different ways, but ultimately both decrease regularization error.

Section 5.2.2 of Goodfellow et al's Deep Learning proposes a much broader definition:

Regularization is any modification we make to a learning algorithm that is intended to reduce its generalization error but not its training error.

There is a tendency to asssociate regularization with shrinkage because of the term "l-p norm regularization"...perhaps "augmentation regularization" is equally valid, although it doesn't roll off the tongue.

Answer 2

User99889 had a good answer (+1), I will add some comments to elaborate the points.

The goal of regularization is reducing the Variance and increase Bias in Bias Variance Trade Off. This can be achieved in different ways, here are some examples:

• Increase the amount of the data, collect new data or derive new data from existing data / data augmentation.
• Reduce the complex of the model. For example, reduce the hidden layers or number of units in the hidden layer.
• Put constraints in the coefficients/parameters. For example, L1 or L2 regularization.

---

We will further to explain why User99889 mentioned about "add prior / domain knowledge". Let use MNIST data as an example, we can create more data by using the following knowledge:

• If we re-scale digit, the label will not be changed
• If we shift digit, the label will not be changed

Then, we have more data using domain knowledge and the model will have more Bias.

Answer 3

The objective of regularization is to improve the generalization capability of the model (in other words its ability to perform well on unseen data). Regularization can be explicit or implicit. The first case refers to techniques that can constrain the effective capacity of the model in order to reduce overfitting. Examples include adding a penalty term in standard statistical estimators such as OLS or weight decay and dropout in deep learning. In the second case (implicit regularization) instead of explicitly constraining the capacity of the model we use indirect methods. One example, is using SGD to train linear models as it always converges to a solution with a small norm (https://arxiv.org/pdf/1611.03530.pdf). Similarly, data augmentation (which has many forms e.g. adding predictors, adding artificially created data, resampling) will help the model to generalize better and is thus considered to be an implicit regularization method.