7
$\begingroup$

Some context is shared below, and my question is bolded at the end.


In the linear regression setting, we learn model weights $\hat{\mathbf{w}}$ to make predictions $\mathbf{\hat{y}}$ from new samples $\mathbf{x}$ as

$$ \hat{\mathbf{y}} = \hat{\mathbf{w}}^T\mathbf{x} $$

When one assumes the true underlying distribution is a linear combination and a Gaussian noise term,

$$ p(\mathbf{y}|\mathbf{x}) = \mathbf{w}^T \mathbf{x} + \mathcal{N}(\mathbf{0}, \Sigma) $$

it's well known that maximum likelihood estimation (MLE) induces a mean squared error loss

$$ \mathcal{L}_{MLE}(\mathbf{\hat{w}}) = \sum_{i=1}^n (\hat{\mathbf{w}}^T\mathbf{x}_i - \mathbf{y})^2 $$

such that minimizing $\mathcal{L}$ produces the MLE estimate of weights.

Further, if one assumes a Gaussian prior distribution on the model weights $\hat{\mathbf{w}}$, then the analogous maximum a posteriori (MAP) estimation induces the L2 regularizer

$$ \mathcal{L}_{MAP}(\mathbf{\hat{w}}) = \sum_{i=1}^n (\hat{\mathbf{w}}^T\mathbf{x}_i - \mathbf{y})^2 + \lambda||\mathbf{\hat{w}}||^2_2 $$

I really appreciate that these common practices (MSE loss and L2 regularization) can be derived from first principles (M-estimators) and simple distributional assumptions (Gaussian observation noise and model prior).

But L2 regularization is used all over the place - practitioners will add an L2 loss on the weights for models of all sorts, from logistic regression to enormous neural networks. I appreciate that it works well, but it seems a bit mysterious. When used outside of the linear regression setting, does L2 regularization still express the same elegant distributional assumptions and first-principles? And a natural follow up question if not, why does it still work so well?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.