Updating posterior probability as more data is given

Question

There are a couple of questions with similar title, but I don't think my doubt is addressed by those, so asking a new question.

I have some random variable $X$ and another $D$ such that initially the prior distribution of $X$ and the conditional distribution of $D$ on $X$ are known. Suppose initially the probability distribution of $X$ is just $P(x)$. If suddenly I receive data on a realization of $D$, say $d_1$, then the distribution of $X$ gets updated as follows:

$$P(x\ |\ d_1)=\frac{P(d_1\ |\ x)P(x)}{P(d_1\ |\ x)P(x)+P(d_1\ |\ x^c)P(x^c)}$$

(assume for now only $x$ and $x^c$ as values for $X$). Now suppose we get new information on another realization of $D$, say $d_2$. How should I write the update equation now? From what I've learned, the old posterior becomes the new prior, so $P(x)$ gets replaced by $P(x\ |\ d_1)$. So the numerator in the expression for $P(x\ |\ d_2, d_1)$ should be $P(d_2\ |\ x)P(x\ |\ d_1)$, is this correct? Should the denominator be $P(d_2\ |\ x)P(x\ |\ d_1)+P(d_2\ |\ x^c)P(x^c\ |\ d_1)$? i.e.

$$P(x\ |\ d_2, d_1)=\frac{P(d_2\ |\ x)P(x\ |\ d_1)}{P(d_2\ |\ x)P(x\ |\ d_1)+P(d_2\ |\ x^c)P(x^c\ |\ d_1)}$$

I understand this makes sense only if "the realizations $d_1$ and $d_2$ occurring are conditionally independent (conditioned on $X$)". Is my phrasing of the quoted part correct? What if there's no conditional independence?

Finally, there's a clear-cut interpretation of the numerator and denominator in the first equation: the numerator is the probability of both $X=x$ and $D=d_1$. The denominator is just probability of $D=d_1$. But what about the second equation? I don't really know how to interpret the numerator and denominator of that one.

Answer 1

Given $d_1$, the Bayes theorem reads,

$$P(x\ |\ d_1)=\frac{P(d_1\ |\ x)P(x)}{P(d_1)}$$

now you get $d_2$, in which case we can write

$$P(x\ |\ d_1, d_2)=\frac{P(d_1, d_2\ |\ x)P(x)}{P(d_1, d_2)}$$

taking into account that $P(d_1, d_2\ |\ x) = P(d_2 \ | \ d_1, \ x)P(d_1 \ | \ x)$ and that $P(d_1, d_2) = P(d_2 \ | \ d_1)P(d_1)$, by substituing we obtain

$$P(x\ |\ d_1, d_2)=\frac{P(d_2 \ | \ d_1, \ x)P(d_1 \ | \ x)P(x)}{P(d_2 \ | \ d_1)P(d_1)}$$

which, using the initial expression for $P(x\ |\ d_1)$, can also be written as follows

$$P(x\ |\ d_1, d_2)=\frac{P(d_2 \ | \ d_1, \ x)P(x\ |\ d_1)P( d_1)}{P(d_2 \ | \ d_1)P(d_1)}$$

which simplifies to

$$P(x\ |\ d_1, d_2)=\frac{P(d_2 \ | \ d_1, \ x)P(x\ |\ d_1)}{P(d_2 \ | \ d_1)}$$

We can see that your posterior distribution $P(x\ |\ d_1)$ already acts as a prior, but your model must take into account how $d_2$ is conditioned on $d_1$. If now we assume that the realizations $d_1$ and $d_2$ are independent, the expression above simplifies to

$$P(x\ |\ d_1, d_2)=\frac{P(d_2 \ | \ x)P(x\ |\ d_1)}{P(d_2)}$$

in which case, the posterior acts again a prior but the different realizations of $D$ are independent of each other.

Regarding the interpretation of the numerators and denominators, I think with my notation is more clear. $P(d_2)$ is just the probability of getting $D=d_2$, independently of anything else whatsoever. $P(d_2 \ | \ d_1)$ is the probability of $D=d_2$ if you already got $D=d_1$ before (such 2 probabilities differ only if the realizations of $D$ are dependent).

Hope it helps.