0
$\begingroup$

There are a couple of questions with similar title, but I don't think my doubt is addressed by those, so asking a new question.

I have some random variable $X$ and another $D$ such that initially the prior distribution of $X$ and the conditional distribution of $D$ on $X$ are known. Suppose initially the probability distribution of $X$ is just $P(x)$. If suddenly I receive data on a realization of $D$, say $d_1$, then the distribution of $X$ gets updated as follows:

$$P(x\ |\ d_1)=\frac{P(d_1\ |\ x)P(x)}{P(d_1\ |\ x)P(x)+P(d_1\ |\ x^c)P(x^c)}$$

(assume for now only $x$ and $x^c$ as values for $X$). Now suppose we get new information on another realization of $D$, say $d_2$. How should I write the update equation now? From what I've learned, the old posterior becomes the new prior, so $P(x)$ gets replaced by $P(x\ |\ d_1)$. So the numerator in the expression for $P(x\ |\ d_2, d_1)$ should be $P(d_2\ |\ x)P(x\ |\ d_1)$, is this correct? Should the denominator be $P(d_2\ |\ x)P(x\ |\ d_1)+P(d_2\ |\ x^c)P(x^c\ |\ d_1)$? i.e.

$$P(x\ |\ d_2, d_1)=\frac{P(d_2\ |\ x)P(x\ |\ d_1)}{P(d_2\ |\ x)P(x\ |\ d_1)+P(d_2\ |\ x^c)P(x^c\ |\ d_1)}$$

I understand this makes sense only if "the realizations $d_1$ and $d_2$ occurring are conditionally independent (conditioned on $X$)". Is my phrasing of the quoted part correct? What if there's no conditional independence?

Finally, there's a clear-cut interpretation of the numerator and denominator in the first equation: the numerator is the probability of both $X=x$ and $D=d_1$. The denominator is just probability of $D=d_1$. But what about the second equation? I don't really know how to interpret the numerator and denominator of that one.

$\endgroup$
3
$\begingroup$

Given $d_1$, the Bayes theorem reads,

$$P(x\ |\ d_1)=\frac{P(d_1\ |\ x)P(x)}{P(d_1)}$$

now you get $d_2$, in which case we can write

$$P(x\ |\ d_1, d_2)=\frac{P(d_1, d_2\ |\ x)P(x)}{P(d_1, d_2)}$$

taking into account that $P(d_1, d_2\ |\ x) = P(d_2 \ | \ d_1, \ x)P(d_1 \ | \ x)$ and that $P(d_1, d_2) = P(d_2 \ | \ d_1)P(d_1)$, by substituing we obtain

$$P(x\ |\ d_1, d_2)=\frac{P(d_2 \ | \ d_1, \ x)P(d_1 \ | \ x)P(x)}{P(d_2 \ | \ d_1)P(d_1)}$$

which, using the initial expression for $P(x\ |\ d_1)$, can also be written as follows

$$P(x\ |\ d_1, d_2)=\frac{P(d_2 \ | \ d_1, \ x)P(x\ |\ d_1)P( d_1)}{P(d_2 \ | \ d_1)P(d_1)}$$

which simplifies to

$$P(x\ |\ d_1, d_2)=\frac{P(d_2 \ | \ d_1, \ x)P(x\ |\ d_1)}{P(d_2 \ | \ d_1)}$$

We can see that your posterior distribution $P(x\ |\ d_1)$ already acts as a prior, but your model must take into account how $d_2$ is conditioned on $d_1$. If now we assume that the realizations $d_1$ and $d_2$ are independent, the expression above simplifies to

$$P(x\ |\ d_1, d_2)=\frac{P(d_2 \ | \ x)P(x\ |\ d_1)}{P(d_2)}$$

in which case, the posterior acts again a prior but the different realizations of $D$ are independent of each other.

Regarding the interpretation of the numerators and denominators, I think with my notation is more clear. $P(d_2)$ is just the probability of getting $D=d_2$, independently of anything else whatsoever. $P(d_2 \ | \ d_1)$ is the probability of $D=d_2$ if you already got $D=d_1$ before (such 2 probabilities differ only if the realizations of $D$ are dependent).

Hope it helps.

$\endgroup$
  • $\begingroup$ Very clear answer. Thanks a lot! So even if $d_1$ and $d_2$ are conditionally independent (conditioned on $X$), the final equation may not hold. It's necessary for the realizations to be independent. Is that correct? $\endgroup$ – Shirish Kulhari Sep 30 '19 at 17:54
  • $\begingroup$ I'm not sure I get your question. My final expression applies if $d_2$ and $d_1$ are independent of each other. If they are not independent of each other, you need my penultimate expression, but that's not all since in that case, you need to know how $d_1$ conditions $d_2$. Maybe you can reword your question and I can help. $\endgroup$ – DavidF Sep 30 '19 at 18:44
  • $\begingroup$ By conditionally independent, I mean $P(d_1, d_2\ |\ x) = P(d_1\ |\ x)P(d_2\ |\ x)$. So I think the simplification of the numerator in your second last equation uses conditional independence, while the simplification of the denominator uses just independence. That's what I was trying to get at. $\endgroup$ – Shirish Kulhari Sep 30 '19 at 19:35
  • $\begingroup$ Conditional independence is $P(d_1,d_2|x)=P(d_1|x) P(d_2|x)$. I don't do that in the numerator, I just use the product rule for joint probability. The relations I use are: $P(d_1, d_2 | x) = P(d_2, d_1 | x)= P(d_2|d_1,x) P(d_1|x)$ and $P(d_1|x)P(x) = P(x|d_1)P(d_1)$. In the denominator we have $P(d_1,d_2)=P(d_1|d_2)P(d_2)$ and then I use independence i.e., $P(d_1,d_2)=P(d_1)P(d_2)$. $\endgroup$ – DavidF Sep 30 '19 at 20:20
  • $\begingroup$ So in the numerator, you have $P(d_2\ |\ d_1, x)=P(d_2,d_1,x)/P(d_1,x)=P(d_2,d_1\ |\ x)P(x)/P(d_1,x)=P(d_2,d_1\ |\ x)/P(d_1\ |\ x)$. For this expression to be equal to $P(d_2\ |\ x)$ as you wrote, we must have $P(d_2,d_1\ |\ x)/P(d_1\ |\ x)=P(d_2\ |\ x)\implies P(d_2,d_1\ |\ x)=P(d_1\ |\ x)P(d_2\ |\ x)$. Does that mean the simplification of the numerator assumes conditional independence? $\endgroup$ – Shirish Kulhari Oct 2 '19 at 8:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.