Updating posterior distribution in online fashion [duplicate]

Question

First, I am sorry if this is an obvious question, I am starting to study bayesian statistics (mainly for machine learning) and I was seeing the classic coin flip example using a Bernoulli distribution with parameter $q$. So I checked the math using an uniform prior ($P(q)=1$), and I was trying to see how the posterior would change after seeing for example $D=\{heads,tails\}$. So After computing the equations, I got that the posterior $P(q|D)=6(q-q^2)$. Then I did the same computation, but this time in 2 steps; that is, first I computed $P(q|D)$ but only with $D=\{heads\}$ which was equal to $2q$. Then I computed the posterior again with $D=\{tails\}$ but this time using the posterior that I just obtained ($P(q)=2q$). And then I obtained the same posterior as before. My question is, is this always the case if we assume iid samples or just works for this simple case? Is this how real life systems are programmed (using the previous posterior distribution as prior)? Thanks!

Answer 1

Say you have some prior $p(x)$ and a likelihood $p(D|x)$ for (say) two iid random variables $y_1, y_2$, so $p(D|x) = p(y_1,y_2|x) = p(y_1|x)p(y_2|x)$ and $p(D) = p(y_1)p(y_2)$. The posterior is then:

$$ p(x|D) = \frac{p(D|x)p(x)}{p(D)} = \frac{p(y_2|x)}{p(y_2)}\frac{p(y_1|x)p(x)}{p(y_1)} . $$

But after seeing $y_1$, we believe a different belief in the distribution of $x$ (by Bayes rule): $$ p(x|y_1) = \frac{p(y_1|x)p(x)}{p(y_1)}, $$

so $$ p(x|D) = \frac{p(y_2|x) p(x|y_1)}{p(y_2)}. $$

But that is just Bayes rule with the prior $p(x|y_1)$!

So your intuition was correct. It does not matter if you imagine you saw all observations at once (as $D$) or one by one (as $y_1,y_2$).

Answer 2

Yair Daon is right, what is also worth mentioning is that what you have observed is called Bayesian updating, and as already mentioned, it can be done all-at-once, in batches, or sequentially. The derivation shown by Yair Daon uses the chain rule in probability and applies not only to the Bayesian inference, but to conditional probabilities in general.