What is the meaning of the variance in a prior that represents L2 regularization?

Question

Why is the L2 regularization equivalent to Gaussian prior? does a nice job in explaining the relationship between L2 regularization and Gaussian priors.

One answer explains that the variance in the Gaussian likelihood is a measure of the noise in the data (which makes intuitive sense).

What is the intuitive understanding of the variance in the Gaussian prior (which is related, by algebraic computations, to the "strength of the regularization")? Based on the above, the variance in the prior is a measure of the "noise" of the "weights of the parametrized function," but that seems an odd and unuseful way to think about it. Is there a better/clear way to understand this variance?

Answer 1

The basic idea of the prior is, that it describes your knowledge about the weights before you get to know the observations (the data). Thus, a prior with a mean equal to zero and a small prior variance keeps the parameters near zero, i.e. it is used as regularization. Alternatively, a large prior variance means that you don't have much knowledge about the weights and the main source of your information are your data. If there is no prior knowledge about the weights, people use uninformative priors, which don't contain information. This could go as far as using improper priors, which are not even proper densities anymore but can still be computed with as far as the purposes of Bayesian inference are concerned.

Note, that the priors are not the only source of the "noise of the weights", i.e. the variance of the posterior, but it is also very much determined by the data.

Furthermore, what is really useful about priors, is that you can learn them, too. You can make (some of) the hyperparameters, that describe the prior, random variables as well, creating a hierarchical model, and also infer them from the data. That way, you could e.g. even create priors that help you learn sparse models, as in the Relevance Vector Machine.

Thus, while the basic purpose of the prior is to convey prior knowledge about your weights, they can have additional purpose if they are learned, too.

Answer 2

In case the prior is a single Gaussian, its variance means how confident you are regarding your prior beliefs. That is, the smaller the variance you put into your prior Gaussian, the more confident you are in your prior belief that the data has to be around the mean of the prior. Calling it "noise", like you say, might be misleading as noise refers to some measurement process, whereas here there is no measurement involved.

Also notice that in the regularization framework, you get the 1/variance in front of the regularization term, which means the bigger your variance, the smaller of a weight your regularization term has, corresponding to your less confident prior belief.

Depending on the situation it can have more meanings. I can give two examples here:

1. if you already have some data which is measured with very very little noise before your experiment, you can use these to estimate the mean and variance of a Gaussian to use as your prior in the inference afterwards. You can call this prior an empirical Gaussian prior. Then the variance would be the variance of your pre-experiment data points.
2. You can extend your model by putting a prior distribution onto the variance of your prior. You can use an inverse gamma distribution for the variance prior, for instance. The Gamma distribution would then have so called hyper-parameters. Then you can control your beliefs regarding the prior by controlling your beliefs regarding its variance, i.e. by playing with the hyper-parameters. Alternatively, you can marginalize over the variance parameter and get rid of it entirely to only speak about the hyper-parameters afterwards.