How is adding noise to training data equivalent to regularization?

Question

I've noticed that some people argue that adding noise to training data equivalent to regularizing our predictor parameters. How is this the case?

1. Some of the examples listed on SE discussing this topic focus more on e.g. LSTMs and SVMs, but can we do this for simpler models like a multiple linear regression?

2. How will it affect our parameters' confidence intervals?

3. Will there be any differences in effects choosing between the various types of white noise, e.g. Gaussian vs uniform white noise?

Answer 1

Adding noise to the regressors in the training data is similar to regularization because it leads to similar results to shrinkage.

The linear regression is an interesting example. Suppose $(Y_i,X_i)_{i=1}^n$ is a set of i.i.d. observations and that $$ Y_i = \beta_0 + \beta_1X_i + U_i \qquad \mathbb{E}[U_i \mid X_i] = 0 $$ The population coefficient for $\beta_1$ is equal to $$ \beta_1 = \frac{Cov(Y_i,X_i)}{Var(X_i)} $$ The estimated OLS coefficient $\hat{\beta}_1$ can be written as a sample analog of $\beta_1$. Now suppose that we add white noise $Z_i = X_i + \varepsilon_i$ and assume that $\mathbb{E}[\varepsilon_i] = 0$, $Var(\varepsilon_i) = \sigma^2$, and that that $\varepsilon_i$ is independent of $Y_i,X_i$. I have made no other assumption about the distribution of $\varepsilon_i$.

Then the population coefficient for a regression of $Y_i$ on $Z_i$ (the noisy regressor) is equal to, $$ \tilde{\beta}_1 = \frac{Cov(Y_i,Z_i)}{Var(Z_i)} = \frac{Cov(Y_i,X_i + \varepsilon_i)}{Var(X_i + \varepsilon_i)} = \frac{Cov(Y_i,X_i)}{Var(X_i) + \sigma^2} = \frac{Var(X_i)}{Var(X_i)+\sigma^2} \times \beta_1 $$ Therefore, $\tilde{\beta}_1$ shrinks to zero for higher values of $\sigma^2$. The estimator for $\tilde{\beta}_1$ will also shrink to zero. We can use the test data to choose a sequence $\sigma_n^2 \to 0$ that achieves the optimal bias-variance trade-off via cross-validation.

If you want to do inference, you clearly need to do some form of adjustment both because the estimator is biased and the variance depends on $\sigma^2$. The process for selecting $\sigma^2$ can also distort the confidence intervals.

Answer 2

Overview: For linear regression, I'll show that $\ell_2$ regularization (a.k.a. ridge regression) arises from minimizing the expected squared error over random perturbations of the regressors. The distributional form of the perturbations doesn't matter beyond some minimal requirements (i.i.d., zero mean). The variance of the perturbations controls the regularization strength.

Let $\big\{(x_i, y_i)\big\}_{i=1}^n$ be the data, with regressors $x_i \in \mathbb{R}^d$ and responses $y_i \in \mathbb{R}$. Suppose we first add random noise to the regressors, then compute predictions as a linear function of the perturbed regressors:

$$\hat{y}_i = (x_i + \delta_i)^T w$$

$w \in \mathbb{R}^d$ are the regression coefficients and the perburbations $\{\delta_i\}$ are i.i.d. random vectors with mean $\vec{0}$ and covariance matrix $\lambda I$. It's not necessary to assume that perturbations are generated from any particular parametric family.

We seek coefficients that minimize the expected squared error $L(w)$, where the expectation is taken over the random perturbations:

$$L(w) = E \left[ \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2 \right]$$

Plug in the above expression for $\hat{y}_i$ and expand:

$$L(w) = E \left[ \frac{1}{n} \sum_{i=1}^n (y_i - x_i^T w)^2 - 2 \delta_i^T w (y_i - x_i^T w) + w^T \delta_i \delta_i^T w \right]$$

By linearity of expectation:

$$L(w) = \frac{1}{n} \sum_{i=1}^n \left( (y_i - x_i^T w)^2 - 2 E \Big[ \delta_i \Big]^T w (y_i - x_i^T w) + w^T E \Big[ \delta_i \delta_i^T \Big] w \right)$$

Note that $E[\delta_i] = \vec{0}$ and $E[\delta_i \delta_i^T] = \lambda I$ are the mean and covariance matrix of the random perturbations:

$$L(w) = \frac{1}{n} \sum_{i=1}^n \left( (y_i - x_i^T w)^2 + \lambda w^T w \right)$$

Simplify, noting that $w^T w = \|w\|_2^2$:

$$L(w) = \frac{1}{n} \sum_{i=1}^n (y_i - x_i^T w)^2 + \lambda \|w\|_2^2$$

This is the mean squared prediction error for the original (non-perturbed) data, plus a penalty on the squared $\ell_2$ norm of the coefficients. Notice that it corresponds exactly to the cost function for ridge regression, with penalty strength $\lambda$.

Answer 3

The basic concept behind regularization is that we start with our Bayesian prior for the coefficients being a decreasing function of the magnitude of the coefficient. That is, the Bayesian prior for the coefficient being large is smaller than the prior for the coefficient being small. If the basic loss function gives a large estimate for the coefficients, our final estimate is a balance between the Bayesian prior giving more weight to smaller estimates, and the data providing evidence for larger ones.

If the random noise corresponds to coefficients being $0$, then it will also pull our final estimates towards being smaller; our actual data saying the coefficients are large will have to compete with the random noise saying the coefficients are small. Another way of looking it is that if we add noise that is generated according our priors, then that will decrease the degree to which our data causes our final estimates of the coefficients to deviate from our priors. Both regularization and random noise are ways of increasing the effects of our priors on our final estimates.