In Bayesian linear regression, why do we assume parameter prior has zero mean?

Question

Bayesian linear regression

$$f(x)=x^Tw$$ $$y=f(x)+\epsilon$$

$w$: is a vector of the parameters $(w_1, w_2,.., w_p)$
$f$: the function value
$y$: observed value
$\epsilon \sim N(0, \sigma^2)$

Prior distribution is $w \sim N(0, \Sigma_p)$. Why is it ok to have zero mean here? I think it's wrong!

This means $x^Tw$ has mean zero, so $f(x)$ is zero. Why would you want to have a model with mean zero? You want to predict $y$!

Unless we are assuming the data $y$ has mean zero, otherwise, this doesn't make sense to me.

Specific examples
Here 1: http://www.gaussianprocess.org/gpml/chapters/RW2.pdf See on page 9
Here 2: https://www.youtube.com/watch?v=dtkGq9tdYcI

Could someone please explain what is going on here?

Answer 1

Encoding assumptions about the data isn't quite the role of the prior distribution in a Bayesian model.

The prior does not reflect any assumptions about the data: It mathematically captures any assumptions the analyst makes about model parameters before observing the data. The posterior distribution reflects both the prior and observed data.

Take this quote from your question:

This means $x^Tw$ has mean zero, so $f(x)$ is zero. Why would you want to have a model with mean zero? You want to predict $y$!

Unless we are assuming the data $y$ has mean zero, otherwise, this doesn't make sense to me.

That $f(x)=0$ is only true using the prior distribution for $w$. It will not be true if one uses the posterior distribution for $w$.

Whether a zero-mean prior accurately encodes your assumptions is another story; but whatever assumptions it does capture do not apply to the data.

Answer 2

You can use any prior that you want. It does not have to be normal, it can have different mean, or it can have no mean (Cauchy)... It is your subjective choice that you make before seeing the data.

Recall that posterior is likelihood times prior

$$ \underbrace{P(\theta|D)}_\text{posterior} \propto \underbrace{P(D|\theta)}_\text{likelihood} \times \underbrace{P(\theta)}_\text{prior} $$

As mentioned by others, prior shows you initial, out-of-data, beliefs about your model and then is updated using observed data. So you can set mean of your prior to zero and if observed data provides enough information so to move it to another value, posterior would be different than zero. This idea is actually often used for purpose, for example Spiegelhalter (2004) describes as different priors may help to test different hypothesis against the data and facilitate decision-making, where zero-mean prior could be used as "sceptical" prior.

As a picture is (sometimes) worth a thousand words, you can check and run yourself one of the examples in JavaScript library bayes.js for MCMC sampling. The example illustrates simple model for estimating $\mu$ and $\sigma$ for normal distribution (i.e. intercept-only regression if you prefer to think of it like this), where model is defined as follows:

$$ x_i \sim \mathrm{Normal}(\mu, \sigma) $$ $$ \mu \sim \mathrm{Normal}(0, 100) $$ $$ \sigma \sim \mathrm{Uniform}(0, 100) $$

You can run the example to convince yourself that not much MCMC iterations are needed for algorithm to converge and for posterior mean to update from prior zero to something around $185$. Prior mean is not what we want posterior mean to be, but what do we think of our model before seeing the data and in many cases before seeing the data you do not know if regression parameters have any effect, i.e. if they are different than zero, so often it is a reasonable choice.

---

Spiegelhalter, D. J. (2004). Incorporating Bayesian ideas into health-care evaluation. Statistical Science, 156-174.

Answer 3

@user13985 IDK what you mean by no prediction. In your particular model you can derive the posterior distribution if you use a normal prior.

If $\mathbf{y}\sim \text{N}(\mathbf{X}\beta,\mathbf{R})$ and $\beta \sim \text{N}(\mathbf{a},\mathbf{B})$ then the posterior is $\beta|\mathbf{y} \sim \text{N}(\mu, \Sigma)$ where $$\mu = \Sigma\left(\mathbf{X}^\intercal\mathbf{R}^{-1}\mathbf{y} + \mathbf{B}^{-1}\mathbf{a}\right)\quad\text{and}\quad\Sigma = \left(\mathbf{X}^\intercal\mathbf{R}^{-1}\mathbf{X} + \mathbf{B}^{-1}\right)^{-1}$$

You can see that the posterior mean is a combination of the observed data $y$ and the prior mean $a$. Using a zero mean places more emphasis on the data $y$, using a non-zero mean shifts the posterior mean. So I don't see what you mean by "destroys the model". The prior is basically your prior belief in the variables, if you don't know anything about the variables, let the data speak for itself by using a zero mean prior.

Answer 4

Perhaps one way to help motivate this is that selecting a normal prior with mean 0 is equivalent to ridge regression (i.e. adding an L2 penalty on your estimated coefficient). Ridge regression has been proven to be helpful in regression, therefore adding a normal prior with mean 0 should be helpful too.