Bayesian regularization intuition - what is a distribution of weights?

Question

I understand the motivation for regularization, as well as its more "conventional" definition as a penalty in the loss function, including how ridge, LASSO and ElasticNet work. But I do not understand Bayesian regularization; specifically, the book I am learning from presents a derivation that starts with defining $\textbf{w} \sim N(0, S_0^{-1}I)$ where $\textbf{w}$ represents the weight vector in our linear regression model, then claims we want to maximize the posterior probability $P(\textbf{w}|\textbf{X}, \textbf{Y}, \beta)$ where $\textbf{X}$ is the matrix of all the input vectors (parameters) and $\textbf{Y}$ the vector of outputs (where each $y_i$ is a scalar output since this is regression).

Then, the book claims

The interpretation of this is that adding a prior over the distribution of our weight parameters and then maximizing the resulting posterior distrinbution is equvalent to adding a regularization term where $\lambda = \frac{S_0}{\beta}$

I am confused about a few things:

1. Broadly, what does it mean for the Bayesian approach to "distribute over weight parameters". I understand it in a hand-wavey way, but I would appreciate if someone could draw an explicit link between what exactly the Bayesian approach is and how it relates to the "optimization of loss function" approach, and connect that with the intuition.
2. Why are we optimizing the posterior? I thought the whole point of the Bayesian approach was to maximize the likelihood, and then choose the weights that gives you that Normal distribution that maximized the likelihood, just as you'd choose the weights by optimization using vector calculus.
3. How does this relate to overfitting? With the conventional "optimization" approach I see how increased values of $\lambda$ increasingly penalize complex weights, and so at the limit you have either a flat line that is totally unresponsive to new training data ($\lambda = \infty$) or a "hardcoded" model that is not generalizable at all ($\lambda = 0$). But how does the Bayesian approach fit into the picture: what does all this trickery have to do with reducing overfitting and penalizing high dimensional weight vectors?

More generally, if someone could give a TL;DR of how the Bayesian approach to linear regression compares to the "optimization" approach and the context and motivation for it as background to their answer, that'd be great.

Answer 1

I will try to answer your questions with following order 2,1,3, I believe understanding question 2 will make it easier to understand remaining questions.

Question 2:

Bayesian inference is done through posterior, not likelihood. The likelihood is just a component of the posterior. Maximizing the likelihood is the frequentist approach since Maximizing the likelihood does not involve prior. Even in the case of Maximizing the likelihood, the purpose is not "choose the weights that give you that Normal distribution." You make the assumption likelihood follows a normal distribution beforehand, and you maximize the likelihood with the assumption you made. But you do not need the assumption of normality; distribution is selected based on the data in hand. But normality provides mathematical simplicity. We are maximizing Posterior because we want to consider both the information data provides and the prior information we have when structuring our model. This is actually

Question 1:

Optimization of loss-function identical to minimizing negative log-likelihood. For instance, mean-squared-error is negative log-likelihood of Gaussian distribution with constant terms dropped. Meaning optimization of a loss function is nothing more than maximizing the likelihood. So what is the likelihood? The likelihood is nothing but a conditional probability of the data you observe. For further understanding, let's review the likelihood of Gaussian distribution with the assumption that mean is a linear combination of weights and features(just linear regression).

$$Y|X, \omega \sim N(X^T \omega, \sigma^2 I)$$

$$p(Y|X, \omega) = \prod_i^N p(y_i|x_i, \omega) = \prod_i^N \frac{1}{\sqrt{2\pi}\sigma}exp\left\{ - \frac{(y_i - x_i^T \omega)^2}{2\sigma^2}\right\}$$

Where:

$Y$: Target variable vector

$\omega$: Weight vector

$\sigma$: standard deviation

$X$: Feature matrix

$I$: Identity matrix

Taking a negative log of the above likelihood and dropping constant terms leads to mean-squared-error. As you can see above, likelihood is just probability calculated through distribution over target variables.

Lets look at the posterior, we obtain post. through Bayes rule given by:

$$p(\omega |Y, X) = \frac{p(Y|X, \omega)p(\omega)}{p(Y|X)}$$

$p(\omega)$ is the prior, lets say our prior is given by:

$$\omega \sim N(0, \Sigma_{\omega})$$

Since prior represents our beliefs about the parameters, it is up to you how to define it.

For simplecity lets remove denominator from posterior and we have:

$$p(\omega |Y, X) \propto p(Y|X, \omega)p(\omega)$$

Now we already calculated first term above, which is likelihood. We defined our prior as a normal distrbution with 0 mean and $\Sigma_{\omega}$ covariance matrix. Then we can calculate posterior by multiplying both terms, we have:

$$p(\omega |Y, X) \propto p(Y|X, \omega) exp\left\{-\frac{1}{2}\Sigma_{\omega}^{-1}(\omega^T \omega) \right\}$$

Where:

$p(\omega) \propto exp\left\{-\frac{1}{2}\Sigma_{\omega}^{-1}(\omega^T \omega) \right\}$

As you can see from the above formulations, the formulation of posterior is similar to likelihood. In fact, the posterior is nothing more than the combination of likelihood and our prior. Thus posterior is a distribution "distributed over the weight parameters," just like likelihood is a distribution "distributed over target variable."

Question 3:

In the formulation of the book you referred $S_0^{-1}I = \Sigma_{\omega}$. As you increase $S_0^{-1}$ you increase the variance of the prior, meaning you increase the range you are searching for the optimal parameter. Meaning you are allowing for more flexibility. As you can realize, as you increase the $S_0^{-1}$, you decrease the $\lambda$, which referred to as punishment term in the penalized maximum likelihood procedure. The formulation I wrote down above in Question 1 is Bayesian linear regression, which corresponds to Ridge regression, and the punishment term $\lambda$ is coming from the quadratic term in the prior($-\frac{1}{2}\Sigma_{\omega}^{-1}(\omega^T \omega)$)