Some context is shared below, and my question is bolded at the end.

In the linear regression setting, we learn model weights $\hat{\mathbf{w}}$ to make predictions $\mathbf{\hat{y}}$ from new samples $\mathbf{x}$ as

$$ \hat{\mathbf{y}} = \hat{\mathbf{w}}^T\mathbf{x} $$

When one assumes the true underlying distribution is a linear combination and a Gaussian noise term,

$$ p(\mathbf{y}|\mathbf{x}) = \mathbf{w}^T \mathbf{x} + \mathcal{N}(\mathbf{0}, \Sigma) $$

it's well known that maximum likelihood estimation (MLE) induces a mean squared error loss

$$ \mathcal{L}_{MLE}(\mathbf{\hat{w}}) = \sum_{i=1}^n (\hat{\mathbf{w}}^T\mathbf{x}_i - \mathbf{y})^2 $$

such that minimizing $\mathcal{L}$ produces the MLE estimate of weights.

Further, if one assumes a Gaussian prior distribution on the model weights $\hat{\mathbf{w}}$, then the analogous maximum a posteriori (MAP) estimation induces the L2 regularizer

$$ \mathcal{L}_{MAP}(\mathbf{\hat{w}}) = \sum_{i=1}^n (\hat{\mathbf{w}}^T\mathbf{x}_i - \mathbf{y})^2 + \lambda||\mathbf{\hat{w}}||^2_2 $$

I really appreciate that these common practices (MSE loss and L2 regularization) can be derived from first principles (M-estimators) and simple distributional assumptions (Gaussian observation noise and model prior).

But L2 regularization is used all over the place - practitioners will add an L2 loss on the weights for models of all sorts, from logistic regression to enormous neural networks. I appreciate that it works well, but it seems a bit mysterious. When used outside of the linear regression setting, does L2 regularization still express the same elegant distributional assumptions and first-principles? And a natural follow up question if not, why does it still work so well?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.