Regularization and Mean Estimation

Question

Suppose I have some i.i.d. data $x_1, \ldots, x_n \sim N(\mu, \sigma^2)$, where $\sigma^2$ is fixed and $\mu$ is unknown, and I want to estimate $\mu$.

Instead of simply giving the MLE of $\mu = \bar{x}$, one could estimate

(1) $\mu = \lambda \mu_0 + (1 - \lambda) \bar{x},$

for some "prior best guess" $\mu_0$. This also has a nice Bayesian interpretation: we place a prior $\mu \sim N(\mu_0, \sigma^2_0)$ on $\mu$, and $\lambda$ is the weighted precision.

I seem to recall that this also has an explicit L2 regularization-interpretation (i.e., we choose some penalty and minimize the squared loss to get the above estimate), similar to things like lasso and ridge regression, but I can't remember how it goes. Can anyone explain what the L2 regularization-interpretation of (1) is?

[More general answers, where the data isn't necessarily normal distributed, are welcome as well.]

Answer 1

Sure, it would be equivalent to the following ridge-like optimization problem:

$\underset{\mu\in\mathbb{R}|\mu_0,\lambda\geq0}{\min} ||x_i-\mu-\mu_0||_2+\lambda\mu^2$

For $\lambda=0$, $\mu+\mu_0$ goes to the OLS solution (i.e. $\bar{x}$), for $\lambda=\infty$, it shrinks to $\mu_0$.

Answer 2

Ridge regression (Hoerl and Kennard, 1988) was initially developed to overcome singularities when inverting $X^tX$ (by adding $\lambda$ to its diagonal elements). Thus, the regularization in this case consists in working with a vc matrix $(X^tX-\lambda I)^{-1}$. This L2 penalization leads to "better" predictions than with usual OLS by optimizing the compromise between bias and variance (shrinkage), but it suffers from considering all coefficients in the model. The regression coefficients are found to be

$$ \hat\beta=\underset{\beta}{\operatorname{argmin}}\|Y-X\beta\|^2 + \lambda\|\beta\|^2 $$

with $\vert\vert\beta\vert\vert^2 = \sum_{j=1}^p\beta_j^2$ (L2-norm).

From a bayesian perspective, you can consider that the $\beta$'s must be small and plug them into a prior distribution. The likelihood $\ell (y,X,\hat\beta,\sigma^2)$ can thus be weighted by the prior probability for $\hat\beta$ (assumed i.i.d. with zero mean and variance $\tau^2$), and the posterior is found to be

$$ f(\beta|y,X,\sigma^2,\tau^2)=(y-\hat\beta^tX)^t(y-\hat\beta^tX)+\frac{\sigma^2}{\tau^2}\hat\beta^t\hat\beta $$

where $\sigma^2$ is the variance of your $y$'s. It follows that this density is the opposite of the residual sum of squares that is to be minimized in the Ridge framework, after setting $\lambda=\sigma^2/\tau^2$.

The bayesian estimator for $\hat\beta$ is thus the same as the OLS one when considering the Ridge loss function with a prior variance $\tau^2$. More details can be found in The Elements of Statistical Learning from Hastie, Tibshirani, and Friedman (§3.4.3, p.60 in the 1st ed.). The second edition is also available for free.