I'm learning bayesian linear regression from the book Bayesian Data Analysis. Here is my questions.

Notation (follow the book): $X$ stands for explanatory variables, with parameter $\psi$; $\bf{y}$ for responsible variables, conditional on $X$ and regression parameter $\theta$.

  1. Why do we need linear regression? It seems we can make predictions on $\tilde{y}$ directly, given the past values of $\bf{y}$. i.e. build a posterior predictive distribution $p(\tilde{y} | \bf{y})$.
  2. In the book, one of the assumptions is (Pg.354 Ch.14):

    The distribution of $X$ is assumed to provide no information about the conditional distribution of $\bf{y}$ given $X$; that is, the parameters $\theta$ determining $p(y|X,\theta)$ and the parameters $\psi$ determining $p(X|\psi)$ are assumed independent in their prior distributions.

    Is this a reasonable assumption? I think we can't say $\psi$ and $\theta$ are independent in the prior, but conditional independent after we observed the variables $X$ and ${y}$. i.e. they are independent in the posterior.

  3. The last question is about the factorization of joint posterior. $$ p(\psi, \theta |X, y) = p(\psi|X)p(\theta|X,y) $$

    According to the author, we can analyze the second factor by itself, with no loss of information: (Pg.355 Ch.14) $$ p(\theta|X,y) \propto p(\theta)p(y|X,\theta) $$

    How to get the above formula? My way: \begin{align} p(\theta|X,y) &= \frac{p(X,y|\theta)p(\theta)}{p(X,y)} \\ &\propto p(X,y|\theta)p(\theta) \end{align}

    In addition, what does the "information loss" refer to?

  1. Yes, you could "forget", that you have explanatory variables, i.e. you could model just the response data $Y$. But why to neglect the information about what influence your response? Suppose you observe a failure times of some device. You can model just those failure times and obtain (maybe) reasonable results. But if you have information about at what age each device failed you could use this information to obtain more accurate predictions, e.g. you have 3 other devices with different ages, then you could predict how much are left for them to operate depending on their specific ages.

  2. This assumption basically means that if we have observed some value of covariates, we don't care anymore about their cause, because this particular observed value we will feed to Y (conditional) distribution to infere about it. Of course, it is just an assumption, very often it makes perfect sense, but it might be that it cannot be stated in other problems. By stating independence of the parameters, we hope that posterior will contain all the information. Assymptotically, i.e. at the infinite size of sample, posterior is independent of prior. But if you have a small sample, then be carefull with stating priors.

  3. You can get it by writing out all the details: the likelihood under stated assumptions is $$p(X,y|\psi, \theta)=p(X|\psi)p(y|\theta,X)$$ Hence: $$p(\psi, \theta|X, y)=\frac{p(X,y|\psi, \theta)p(\psi, \theta)}{p(X,y)}=\frac{p(X|\psi)p(y|\theta,X)p(\theta)p(\psi)}{p(X)p(y|X)}=p(\psi|X)p(\theta|y,X)$$

Since $p(X)$ and $p(y|X)$ are just constant, we can write (up to a constant) $$p(\psi, \theta|X, y)\propto p(X|\psi)p(y|\theta, X)p(\theta)p(\psi) $$ Or $$p(\psi|X)\propto p(X|\psi)p(\psi) $$ $$p(\theta|y,X)\propto p(y|\theta,X)p(\theta) $$

p.s. Of course, first two questions require much more extensive answers, than I just gave. But, hopefully, I shed some light on the issue.

  • 1
    $\begingroup$ In your 3rd answer "the likelihood under stated assumptions" What assumptions here to make the likelihood factorizable? $\endgroup$ – fishiwhj Feb 8 '13 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.