Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Bayesian linear regression

$$f(x)=x^Tw$$ $$y=f(x)+\epsilon$$

$w$: is a vector of the parameters $(w_1, w_2,.., w_p)$
$f$: the function value
$y$: observed value
$\epsilon \sim N(0, \sigma^2)$

Prior distribution is $w \sim N(0, \Sigma_p)$. Why is it ok to have zero mean here? I think it's wrong!

This means $x^Tw$ has mean zero, so $f(x)$ is zero. Why would you want to have a model with mean zero? You want to predict $y$!

Unless we are assuming the data $y$ has mean zero, otherwise, this doesn't make sense to me.

Specific examples
Here 1: http://www.gaussianprocess.org/gpml/chapters/RW2.pdf See on page 9
Here 2: https://www.youtube.com/watch?v=dtkGq9tdYcI

Could someone please explain what is going on here?

share|improve this question
2  
You can use any prior that you want. It does not have to be normal, it can have different mean, or it can have no mean (Cauchy)... – Tim Jan 9 at 19:03
    
@Tim It doesn't make sense why you would EVER use normal zero mean? There is no prediction with zero mean. It destroys the model. – user13985 Jan 9 at 19:07
1  
Posterior is prior times likelihood, so your prior is modified by what your data tells. Zero-mean prior does not have to leave to zero-mean posterior -- if it does it suggests that your data is not enough "informative" to influence it. – Tim Jan 9 at 19:43
4  
You could also think of a zero-mean prior as a conservative choice. If the data pushes the estimate away from zero, it will be purely due to the data, not due to some initial preconception. In a way, you do not allow yourself to bias the result towards what you want to find (unless you want the estimate to be zero, which is not that common). – Richard Hardy Jan 9 at 21:32
2  
What mean value would you prefer the prior to have? – Glen_b Jan 10 at 3:41

It would seem that you aren't yet fully familiar with the role of a prior distribution in a Bayesian model.

The prior does not reflect any assumptions about the data: It mathematically captures any assumptions the analyst makes about model parameters before observing the data. The posterior distribution reflects both the prior and observed data.

Take this quote from your question:

This means $x^Tw$ has mean zero, so $f(x)$ is zero. Why would you want to have a model with mean zero? You want to predict $y$!

Unless we are assuming the data $y$ has mean zero, otherwise, this doesn't make sense to me.

That $f(x)=0$ is only true using the prior distribution for $w$. It will not be true if one uses the posterior distribution for $w$.

Whether a zero-mean prior accurately encodes your assumptions is another story; but whatever assumptions it does capture do not apply to the data.

share|improve this answer

@user13985 IDK what you mean by no prediction. In your particular model you can derive the posterior distribution if you use a normal prior.

If $\mathbf{y}\sim \text{N}(\mathbf{X}\beta,\mathbf{R})$ and $\beta \sim \text{N}(\mathbf{a},\mathbf{B})$ then the posterior is $\beta|\mathbf{y} \sim \text{N}(\mu, \Sigma)$ where $$\mu = \Sigma\left(\mathbf{X}^\intercal\mathbf{R}^{-1}\mathbf{y} + \mathbf{B}^{-1}\mathbf{a}\right)\quad\text{and}\quad\Sigma = \left(\mathbf{X}^\intercal\mathbf{R}^{-1}\mathbf{X} + \mathbf{B}^{-1}\right)^{-1}$$

You can see that the posterior mean is a combination of the observed data $y$ and the prior mean $a$. Using a zero mean places more emphasis on the data $y$, using a non-zero mean shifts the posterior mean. So I don't see what you mean by "destroys the model". The prior is basically your prior belief in the variables, if you don't know anything about the variables, let the data speak for itself by using a zero mean prior.

share|improve this answer

You can use any prior that you want. It does not have to be normal, it can have different mean, or it can have no mean (Cauchy)... It is your subjective choice that you make before seeing the data.

Recall that posterior is likelihood times prior

$$ \underbrace{P(\theta|D)}_\text{posterior} \propto \underbrace{P(D|\theta)}_\text{likelihood} \times \underbrace{P(\theta)}_\text{prior} $$

As mentioned by others, prior shows you initial, out-of-data, beliefs about your model and then is updated using observed data. So you can set mean of your prior to zero and if observed data provides enough information so to move it to another value, posterior would be different than zero. This idea is actually often used for purpose, for example Spiegelhalter (2004) describes as different priors may help to test different hypothesis against the data and facilitate decision-making, where zero-mean prior could be used as "sceptical" prior.

As a picture is (sometimes) worth a thousand words, you can check and run yourself one of the examples in JavaScript library bayes.js for MCMC sampling. The example illustrates simple model for estimating $\mu$ and $\sigma$ for normal distribution (i.e. intercept-only regression if you prefer to think of it like this), where model is defined as follows:

$$ x_i \sim \mathrm{Normal}(\mu, \sigma) $$ $$ \mu \sim \mathrm{Normal}(0, 100) $$ $$ \sigma \sim \mathrm{Uniform}(0, 100) $$

You can run the example to convince yourself that not much MCMC iterations are needed for algorithm to converge and for posterior mean to update from prior zero to something around $185$. Prior mean is not what we want posterior mean to be, but what do we think of our model before seeing the data and in many cases before seeing the data you do not know if regression parameters have any effect, i.e. if they are different than zero, so often it is a reasonable choice.


Spiegelhalter, D. J. (2004). Incorporating Bayesian ideas into health-care evaluation. Statistical Science, 156-174.

share|improve this answer

Perhaps one way to help motivate this is that selecting a normal prior with mean 0 is equivalent to ridge regression (i.e. adding an L2 penalty on your estimated coefficient). Ridge regression has been proven to be helpful in regression, therefore adding a normal prior with mean 0 should be helpful too.

share|improve this answer
    
Is this strictly true of a zero-mean prior, without other restrictions? I thought the prior also specified a scaled identity matrix for covariance, see e.g. here. – Sean Easter Jan 10 at 17:16
    
@SeanEaster: I believe you are asking if ridge regression is equivalent to a prior with a scaled identity matrix for the covariance? I believe that this is true if $\lambda_i$, the ridge penalty, is constant for all $\beta_i$. If you allow $\lambda_i$ to change with $i$, this should be equivalent to having different values on the diagonal. – Cliff AB Jan 10 at 20:40
    
That's actually a step further than I intended, though I'm pleased to learn it. I interpreted your answer to mean that any normal, zero-mean prior on the coefficients is equivalent to ridge regression, irrespective of the prior on the covariance parameters. Is that the case, or does one need to further specify $\Sigma$, either as a scaled identity or diagonal matrix? – Sean Easter Jan 10 at 21:13
    
@SeanEaster: if the $\Sigma$ is not a diagonal matrix, it would not be equivalent to a standard ridge regression method as far as I know. Perhaps if ridge penalty was more complicated, i.e. includes interactions between different coefficients. But I'm only speculating at this point. – Cliff AB Jan 10 at 21:32

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.