2
$\begingroup$

I'm having some trouble following the logic of this passage from Chapter 14 in Bayesian Data Analysis, A. Gelman:

The numerical 'data' in a regression problem includes both $X$ and $y$. Thus, a full Bayesian model includes a distribution for $X$, $p(X|\psi$), indexed by a parameter vector $\psi$, and thus involves a joint likelihood $p(X,y|\psi,\theta)$, along with a prior distribution, $p(\psi,\theta)$. In the standard regression context, the distribution of $X$ is assumed to provide no information about the conditional distribution of $y$ given $X$; that is, we assume prior independence of parameters $\theta$ determining $p(y|X,\theta)$ and the parameters $\psi$ determining $p(X|\psi).$

Thus, from a Bayesian perspective, the defining characteristic of a 'regression model' is that it ignores the information supplied by $X$ about ($\psi$, $\theta$). How can this be justified? Suppose $\psi$ and $\theta$ are independent in their prior distribution; that is $p(\theta,\psi) = p(\theta)p(\psi)$. Then the posterior distribution factors,

$p(\psi,\theta|X,y) = p(\psi|X)p(\theta|X,y)$, [...]

When I work this out I can't obtain the last line. I can get

$p(\psi,\theta|X,y) = p(\psi|X,y,\theta)p(\theta|X,y)$.

Intuitively the statement makes sense, but I can't prove to myself that it is true.

$\endgroup$
1
$\begingroup$

Once you've conditioned on $X$ there is no further information in $Y$ or $\theta$ concerning $\psi$, so $p(\psi|X,\theta,y) = p(\psi|X)$. This is a consequence of the independent priors on $\theta$ and $\psi$.

Concretely, in the full conditional distribution for $\psi$, which is proportional to the full joint distribution, you can factor out $p(y|X,\theta)$ and $p(\theta)$: $$p(\psi|X, \theta,y) \propto p(\psi)\;p(\theta)\;p(X|\psi)\;p(Y|X,\theta) \propto p(\psi) \;p(X|\psi)\propto p(\psi|X)$$

$\endgroup$
  • $\begingroup$ I see. I wasn't taking advantage of the fact that the other terms were normalization constants. $\endgroup$ – PeteyCoco May 1 at 19:22
2
$\begingroup$

It would help here drawing a DAG (or causal diagram) representing the dependencies among the random variables $\psi,\theta,X,y$. It would be $$ \psi \rightarrow X \rightarrow y \\ ~~~~~~~ \theta \nearrow $$Referring to this diagram will help with the calculations.

Then we can do the calculations $$\begin{align} p(\psi,\theta \mid X,y)&=&\frac{p(X,y\mid \psi,\theta) p(\psi)p(\theta)}{p(X,y)}\\ &=& \frac{p(y\mid X,\psi,\theta)\cdot p(X\mid\psi)p(\psi)p(\theta)}{p(y\mid X)p(X)} \\ &=& \frac{p(X\mid\psi)p(\psi)}{p(X)}\cdot \frac{p(y\mid X,\psi,\theta)p(\theta)}{p(y\mid X)} \\ &=& p(\psi\mid X)\cdot \frac{p(y,X,\psi,\theta)p(\theta)/ p(X,\psi,\theta)}{p(y\mid X)} \\ &=& p(\psi\mid X)\cdot\frac{p(\theta\mid y,X,\psi)p(y,X,\psi)p(\theta)}{p(y\mid X)p(X,\psi\mid \theta)p(\theta)}\\ &=& p(\psi\mid X)\cdot\frac{p(\theta\mid y,X,\psi)p(y\mid X,\psi)}{p(y\mid X)}\\ &=& p(\psi\mid X)\cdot p(\theta \mid X,y) \end{align}$$ This could be compared with the frequentist argument in What is the difference between conditioning on regressors vs. treating them as fixed?.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.