I understand the motivation for regularization, as well as its more "conventional" definition as a penalty in the loss function, including how ridge, LASSO and ElasticNet work. But I do not understand Bayesian regularization; specifically, the book I am learning from presents a derivation that starts with defining $\textbf{w} \sim N(0, S_0^{-1}I)$ where $\textbf{w}$ represents the weight vector in our linear regression model, then claims we want to maximize the posterior probability $P(\textbf{w}|\textbf{X}, \textbf{Y}, \beta)$ where $\textbf{X}$ is the matrix of all the input vectors (parameters) and $\textbf{Y}$ the vector of outputs (where each $y_i$ is a scalar output since this is regression).
Then, the book claims
The interpretation of this is that adding a prior over the distribution of our weight parameters and then maximizing the resulting posterior distrinbution is equvalent to adding a regularization term where $\lambda = \frac{S_0}{\beta}$
I am confused about a few things:
- Broadly, what does it mean for the Bayesian approach to "distribute over weight parameters". I understand it in a hand-wavey way, but I would appreciate if someone could draw an explicit link between what exactly the Bayesian approach is and how it relates to the "optimization of loss function" approach, and connect that with the intuition.
- Why are we optimizing the posterior? I thought the whole point of the Bayesian approach was to maximize the likelihood, and then choose the weights that gives you that Normal distribution that maximized the likelihood, just as you'd choose the weights by optimization using vector calculus.
- How does this relate to overfitting? With the conventional "optimization" approach I see how increased values of $\lambda$ increasingly penalize complex weights, and so at the limit you have either a flat line that is totally unresponsive to new training data ($\lambda = \infty$) or a "hardcoded" model that is not generalizable at all ($\lambda = 0$). But how does the Bayesian approach fit into the picture: what does all this trickery have to do with reducing overfitting and penalizing high dimensional weight vectors?
More generally, if someone could give a TL;DR of how the Bayesian approach to linear regression compares to the "optimization" approach and the context and motivation for it as background to their answer, that'd be great.