0
$\begingroup$

I'm going through the MIT OCW notes on probability and statistics and came across an example in some class notes where the posterior is updated after two events: $x_1 = 1$ and $x_2=1$, which correspond to a coin landing on heads twice. To explain how to calculate the posterior $p(\theta|x_1=1, x_2=1)$, the notes say

In life we are continually updating our beliefs with each new experience of the world. In Bayesian inference, after updating the prior to the posterior, we can take more data and update again! For the second update, the posterior from the first data becomes the prior for the second data.

I intuitively how this could make sense, but I don't mathematically understand why we can calculate $$p(\theta|x_1=1, x_2=1) = \frac{p(x_2=1|\theta)p(x_1=1|\theta)p(\theta)}{\alpha}$$

where $\alpha$ is some normalizing factor so that the sum of probabilities equals 1.

Thus, my question is: is there a proof for why can we update the posterior more than one time? Under what conditions can we do this?

$\endgroup$
  • 2
    $\begingroup$ This might not need a mathematical demonstration. After updating the posterior once, which prior would you want to use for the next update? The one that Bayesian theory tells you is the right one or something else? $\endgroup$ – whuber May 5 at 19:54
3
$\begingroup$

Assume that $X_1,..., X_n$ arrive sequentially.

How can we update our inference on $\theta$, given that we start with prior $p(\theta)$?

At time step $n$ we know the likelihood $$ p(x_1,...,x_n|\theta) = p(x_1| \theta)p(x_2 | x_1, \theta),..., p(x_n | x_{1:n−1}, θ) $$ After observing $n$ points we can compute the posterior distribution $$ p(\theta|x_{1:n}) \propto p(\theta) p(x_{1:n}|\theta). $$

Now, assume that $x_{n+1}$ is available. We can compute the new posterior distribution in the same way as before, starting with the prior and including all $n+1$ points: $$ p(\theta|x_{1:n+1}) \propto p(\theta)p(x_{1:n+1}|\theta). $$

Another way would be to consider $p(\theta|x_{1:n})$ as the current prior on $\theta$ and update this prior with the new observation:

$$ p(\theta|x_{1:n+1}) \propto p(\theta) p(x_{1:n}|\theta)p(x_{n+1}|x_{1:n},\theta) =p(\theta)p(x_{1:n+1}|\theta). $$

This approach provides a simple way to update the posterior distribution, also called the current "state of knowledge", in an online fashion, after arrival of each single data point.

The two approaches yield the same posterior distribution.

$\endgroup$
  • $\begingroup$ Hi @dnqxt, quick follow-up question: how did you go from $p(x_{1:n}|\theta)p(x_{n+1}|x_{1:n}, \theta)$ to $p(x_{1:n+1}|\theta)$? Did you assume conditional independence between $x_{1:n}$ and $x_{n+1}$ ergo $p(x_{n+1}|x_{1:n}, \theta) = p(x_{n+1}|\theta)$ followed by using conditional independence between $x_{1:n}$ and $x_{n+1}$ again, ergo $p(x_{1:n+1}|\theta) = p(x_{1:n}|\theta)p(x_{n+1}|\theta)$? Thus, is conditional independence between $x_{1:n}$ and $x_{n+1}$ necessary to yield the same posterior distribution? $\endgroup$ – Noah Stebbins May 6 at 23:48
  • $\begingroup$ That's a good question, but the answer is no, $x_i$-s are not assumed conditionally independent given the initial $\theta$. $\theta$ changes at each stage if one adopts the second (step by step or online) approach above. That's why it's necessary to include all $x_i$-s in the "batch" likelihood in the first approach according to the product rule. As for conditional independence of $x_i$-s given $\theta$ that indeed shows up in a popular Naive Bayes model, but that's another topic. $\endgroup$ – dnqxt May 7 at 0:09
  • $\begingroup$ Hey @dnqxt, I'm not quite sure I follow. If the $x_i$-s are not assumed conditionally independent, then how does $p(x_{1:n}|\theta)p(x_{n+1}|x_{1:n}, \theta) = p(x_{1:n+1}|\theta)$ in the second approach? $\endgroup$ – Noah Stebbins May 7 at 0:25
  • $\begingroup$ Where do you see independence in that equality? $\endgroup$ – dnqxt May 7 at 0:42
  • $\begingroup$ Hi @Noah Stebbins -- how about accepting the answer? Accepting it makes the question easier to find to whoever may look for an answer in the future and will also bring me whopping 15 points :)) $\endgroup$ – dnqxt May 10 at 2:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.