5
$\begingroup$

Suppose $X$ and $Y$ are independent random variables with prior distributions $p_X(x)$ and $p_Y(y)$. Suppose now that we have data $D$ that are characterized by $X,Y$ and that after observing the data, we have a posterior distribution:

$$ p(X,Y|D) $$

Why is it that two independent priors do not result in a posterior distribution that is also independent, or that $p(X,Y|D) \neq p(X|D)p(Y|D)$?

Is there a further implication here in terms of real-life applications? Thanks.

$\endgroup$
8
$\begingroup$

When you have an independent prior on $X$ and $Y$, then the posterior might not factor into $X$ and $Y$ pieces just because the likelihood doesn't factor into $X$ and $Y$ pieces.

It's easy to see that $$ p(x,y|D) \propto p(D|X,Y)p_X(x)p_Y(y). $$ So in your situation the posterior factors if and only if the likelihood factors.

$\endgroup$
4
  • $\begingroup$ Can the likelihood ever even factor? I say this because we have that $p(D|X,Y) = \frac{p(D,X,Y)}{p(X,Y)}$ and so I don't see how we can ever get $p(D|X)p(D|Y)$ as this would require two "copies" of $D$? $\endgroup$
    – user321627
    Jun 26 '17 at 5:27
  • $\begingroup$ @user321627 you might be right I can't think of any examples $\endgroup$
    – Taylor
    Jun 29 '17 at 3:47
  • 3
    $\begingroup$ @user321627 I can't think of any univariate examples, but $D$ could be a 2-d vector of conditionally independent random variables, each dependent on only one of the parameters. $p(d|x,y) = p(d_1|x,y)p(d_2|x,y) = p(d_1|x)p(d_2|y)$ $\endgroup$
    – Taylor
    Jun 29 '17 at 13:20
  • $\begingroup$ @user321627 your reasoning is flawed -- $p(D|X,Y) =p(D|X)p(D|Y)$ does have solutions -- $\frac{P(D,X,Y)}{P(X,Y)} = \frac{P(D,X)}{P(X)}\frac{P(D,Y)}{P(Y)} \implies P(D,X,Y) = P(D,X)P(D,Y)$. Any joint distribution that follows this last factorization works. You do not have to worry about there being two Ds because there is no linearity in the probability distribution function. i.e $P(X)$ is not$\propto X $ $\endgroup$ Apr 29 '18 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.