Does order of events matter in Bayesian update?

Question

I'm wondering whether the order of events can lead to different Bayesian update.

For example, consider a coin-tossing problem with unknown $p$, the probability of Head. Initially, $p$ is known to follow some beta distribution: $$p\sim Beta(a_0,b_0).$$ Suppose that we have a sequence of observations that do not have to be an outcome of coin-tossing. For example, the first observation is "$\mathbb E[p]>\frac{1}{2}$" while the second observation is "Head".

If I want to update $p$ using Baye's rule, it will be a lot easier if I can process the second event first and then the first event later as Beta is a conjugate prior of binomial experiments.

However, if I have to update $p$ in the order of the events (first observation first, and then the second one later), the process requires a bit more of computation.

So, my question is that does the order of events matter in Bayesian updating? If not, what can be a theoretical background that justifies it?

Answer 1

AFAIK, you cannot say that $p > \frac{1}{2}$ or even $\mathbb E[p]>\frac{1}{2}$ is an "observation" or "event", but rather a constraint on your model parameter(s). The term "observation" is usually reserved for specific realizations of a random variable (i.e. draws from a distribution). In your model, one could plausibly observe $p$ (numbers between 0 and 1) or the outcomes of $Bernoulli(p)$ (either 0 or 1). There is no way to observe $\mathbb E[p]>\frac{1}{2}$, this information lies outside your probabilistic model.

As a rough rule (some caveats apply) you should be able to simulate observations from your probabilistic model, given model parameters. How would you simulate a model where $\mathbb E[p]>\frac{1}{2}$ is a possible observation?

If you start with the $p\sim Beta(a_0,b_0); a_0, b_0 \in \mathbb{R}^+$ model and then learn that $\mathbb E[p]>\frac{1}{2}$, it means your initial model was incorrect and you should change your model to reflect the constraint ($\mathbb E[p]>\frac{1}{2}$ implies $\frac{a_0}{a_0 + b_0} > \frac{1}{2}$, so some combinations of $a_0$ and $b_0$ are ruled out). Adding a constraint cannot be AFAIK directly handled in the language of Bayesian updating.

Hope that helps.

Answer 2

In Bayesian inference, terms like "observation" and "event" are just conveniences; there is no fundamental importance to them, so don't get hung up on them.

In particular, there is no physical causality or time's arrow -- no "events". Whether you can carry out some calculations in more than one order depends solely on the form of the model. If, algebraically, the results are the same assuming different orders of assignments to some variables (i.e., "observations"), then, terrific, you can do whatever is convenient. If not, well, so what?

About the representation of p > 1/2, you could represent that as a likelihood function which is just a step at 1/2. That is, it is zero to the left of 1/2, and any positive constant to the right. Note that ordinary "observations" yield likelihood functions which vary smoothly, but the smoothness is not a requirement.

Answer 3

In order for this to be the case, the random variables must be exchangeable.

Your example is a little different since $p>1/2$ isn't an event. An event should be in the support of the likelihood. In this case, events are only constituted by binomial random variables, or sums thereof.