What mathematically enables one to Bayesian update multiple times?

Question

I'm going through the MIT OCW notes on probability and statistics and came across an example in some class notes where the posterior is updated after two events: $x_1 = 1$ and $x_2=1$, which correspond to a coin landing on heads twice. To explain how to calculate the posterior $p(\theta|x_1=1, x_2=1)$, the notes say

In life we are continually updating our beliefs with each new experience of the world. In Bayesian inference, after updating the prior to the posterior, we can take more data and update again! For the second update, the posterior from the first data becomes the prior for the second data.

I intuitively how this could make sense, but I don't mathematically understand why we can calculate $$p(\theta|x_1=1, x_2=1) = \frac{p(x_2=1|\theta)p(x_1=1|\theta)p(\theta)}{\alpha}$$

where $\alpha$ is some normalizing factor so that the sum of probabilities equals 1.

Thus, my question is: is there a proof for why can we update the posterior more than one time? Under what conditions can we do this?

Answer 1

Assume that $X_1,..., X_n$ arrive sequentially.

How can we update our inference on $\theta$, given that we start with prior $p(\theta)$?

At time step $n$ we know the likelihood $$ p(x_1,...,x_n|\theta) = p(x_1| \theta)p(x_2 | x_1, \theta),..., p(x_n | x_{1:n−1}, θ) $$ After observing $n$ points we can compute the posterior distribution $$ p(\theta|x_{1:n}) \propto p(\theta) p(x_{1:n}|\theta). $$

Now, assume that $x_{n+1}$ is available. We can compute the new posterior distribution in the same way as before, starting with the prior and including all $n+1$ points: $$ p(\theta|x_{1:n+1}) \propto p(\theta)p(x_{1:n+1}|\theta). $$

Another way would be to consider $p(\theta|x_{1:n})$ as the current prior on $\theta$ and update this prior with the new observation:

$$ p(\theta|x_{1:n+1}) \propto p(\theta) p(x_{1:n}|\theta)p(x_{n+1}|x_{1:n},\theta) =p(\theta)p(x_{1:n+1}|\theta). $$

This approach provides a simple way to update the posterior distribution, also called the current "state of knowledge", in an online fashion, after arrival of each single data point.

The two approaches yield the same posterior distribution.