I've noticed that some people argue that adding noise to training data equivalent to regularizing our predictor parameters. How is this the case?

  1. Some of the examples listed on SE discussing this topic focus more on e.g. LSTMs and SVMs, but can we do this for simpler models like a multiple linear regression?

  2. How will it affect our parameters' confidence intervals?

  3. Will there be any differences in effects choosing between the various types of white noise, e.g. Gaussian vs uniform white noise?


Adding noise to the regressors in the training data is similar to regularization because it leads to similar results to shrinkage.

The linear regression is an interesting example. Suppose $(Y_i,X_i)_{i=1}^n$ is a set of i.i.d. observations and that $$ Y_i = \beta_0 + \beta_1X_i + U_i \qquad \mathbb{E}[U_i \mid X_i] = 0 $$ The population coefficient for $\beta_1$ is equal to $$ \beta_1 = \frac{Cov(Y_i,X_i)}{Var(X_i)} $$ The estimated OLS coefficient $\hat{\beta}_1$ can be written as a sample analog of $\beta_1$. Now suppose that we add white noise $Z_i = X_i + \varepsilon_i$ and assume that $\mathbb{E}[\varepsilon_i] = 0$, $Var(\varepsilon_i) = \sigma^2$, and that that $\varepsilon_i$ is independent of $Y_i,X_i$. I have made no other assumption about the distribution of $\varepsilon_i$.

Then the population coefficient for a regression of $Y_i$ on $Z_i$ (the noisy regressor) is equal to, $$ \tilde{\beta}_1 = \frac{Cov(Y_i,Z_i)}{Var(Z_i)} = \frac{Cov(Y_i,X_i + \varepsilon_i)}{Var(X_i + \varepsilon_i)} = \frac{Cov(Y_i,X_i)}{Var(X_i) + \sigma^2} = \frac{Var(X_i)}{Var(X_i)+\sigma^2} \times \beta_1 $$ Therefore, $\tilde{\beta}_1$ shrinks to zero for higher values of $\sigma^2$. The estimator for $\tilde{\beta}_1$ will also shrink to zero. We can use the test data to choose a sequence $\sigma_n^2 \to 0$ that achieves the optimal bias-variance trade-off via cross-validation.

If you want to do inference, you clearly need to do some form of adjustment both because the estimator is biased and the variance depends on $\sigma^2$. The process for selecting $\sigma^2$ can also distort the confidence intervals.

  • 10
    $\begingroup$ In pictures: suppose your data is in form of a sequence of points along a continuous but nowhere smooth curve. If you add noise, the curve thickens and a possibility emerges to approximate with a smooth line staying inside that thickened curve. $\endgroup$ Oct 19 '21 at 4:34

Overview: For linear regression, I'll show that $\ell_2$ regularization (a.k.a. ridge regression) arises from minimizing the expected squared error over random perturbations of the regressors. The distributional form of the perturbations doesn't matter beyond some minimal requirements (i.i.d., zero mean). The variance of the perturbations controls the regularization strength.

Let $\big\{(x_i, y_i)\big\}_{i=1}^n$ be the data, with regressors $x_i \in \mathbb{R}^d$ and responses $y_i \in \mathbb{R}$. Suppose we first add random noise to the regressors, then compute predictions as a linear function of the perturbed regressors:

$$\hat{y}_i = (x_i + \delta_i)^T w$$

$w \in \mathbb{R}^d$ are the regression coefficients and the perburbations $\{\delta_i\}$ are i.i.d. random vectors with mean $\vec{0}$ and covariance matrix $\lambda I$. It's not necessary to assume that perturbations are generated from any particular parametric family.

We seek coefficients that minimize the expected squared error $L(w)$, where the expectation is taken over the random perturbations:

$$L(w) = E \left[ \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2 \right]$$

Plug in the above expression for $\hat{y}_i$ and expand:

$$L(w) = E \left[ \frac{1}{n} \sum_{i=1}^n (y_i - x_i^T w)^2 - 2 \delta_i^T w (y_i - x_i^T w) + w^T \delta_i \delta_i^T w \right]$$

By linearity of expectation:

$$L(w) = \frac{1}{n} \sum_{i=1}^n \left( (y_i - x_i^T w)^2 - 2 E \Big[ \delta_i \Big]^T w (y_i - x_i^T w) + w^T E \Big[ \delta_i \delta_i^T \Big] w \right)$$

Note that $E[\delta_i] = \vec{0}$ and $E[\delta_i \delta_i^T] = \lambda I$ are the mean and covariance matrix of the random perturbations:

$$L(w) = \frac{1}{n} \sum_{i=1}^n \left( (y_i - x_i^T w)^2 + \lambda w^T w \right)$$

Simplify, noting that $w^T w = \|w\|_2^2$:

$$L(w) = \frac{1}{n} \sum_{i=1}^n (y_i - x_i^T w)^2 + \lambda \|w\|_2^2$$

This is the mean squared prediction error for the original (non-perturbed) data, plus a penalty on the squared $\ell_2$ norm of the coefficients. Notice that it corresponds exactly to the cost function for ridge regression, with penalty strength $\lambda$.

  • $\begingroup$ How did you come to the result of E[δiδTi]=λI ? I mean how it can be a diagonal matrix ? Is it that expected covariance of different noise coming from same distribution = 0 ?Please elaborate or provide some links. $\endgroup$ Dec 1 '21 at 22:12
  • $\begingroup$ @GadaaDhaariGeek I began by defining each $\delta_i$ as a random vector with mean zero and covariance matrix $\lambda I$. Because $\delta_i$ has mean zero, $E[\delta_i \delta_i^T]$ is its covariance matrix, which is equal to $\lambda I$ by definition. $\endgroup$
    – user20160
    Dec 2 '21 at 0:15
  • $\begingroup$ My question is, why the covariance matrix is a diagonal matrix ? $\endgroup$ Dec 2 '21 at 2:15
  • $\begingroup$ Okay, I think I got it. Since, the noise vectors is independent and identically distributed, their covariance matrix will only have diagonal entries, which is in fact variance of their distribution and other entries as 0. Also, since they are identically distributed their variance is all same. Please, correct me if I am wrong. $\endgroup$ Dec 2 '21 at 9:16
  • $\begingroup$ @GadaaDhaariGeek Not quite. $\delta_i$ is a random vector representing the noise added to the $i$th point along each input dimension (each point has its own $\delta_i$). Defining the covariance matrix of $\delta_i$ as $\lambda I$ means the noise added to each dimension is uncorrelated, and has equal variance in all directions. $\endgroup$
    – user20160
    Dec 2 '21 at 18:55

The basic concept behind regularization is that we start with our Bayesian prior for the coefficients being a decreasing function of the magnitude of the coefficient. That is, the Bayesian prior for the coefficient being large is smaller than the prior for the coefficient being small. If the basic loss function gives a large estimate for the coefficients, our final estimate is a balance between the Bayesian prior giving more weight to smaller estimates, and the data providing evidence for larger ones.

If the random noise corresponds to coefficients being $0$, then it will also pull our final estimates towards being smaller; our actual data saying the coefficients are large will have to compete with the random noise saying the coefficients are small. Another way of looking it is that if we add noise that is generated according our priors, then that will decrease the degree to which our data causes our final estimates of the coefficients to deviate from our priors. Both regularization and random noise are ways of increasing the effects of our priors on our final estimates.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.