4
$\begingroup$

I'm wondering whether the order of events can lead to different Bayesian update.

For example, consider a coin-tossing problem with unknown $p$, the probability of Head. Initially, $p$ is known to follow some beta distribution: $$p\sim Beta(a_0,b_0).$$ Suppose that we have a sequence of observations that do not have to be an outcome of coin-tossing. For example, the first observation is "$\mathbb E[p]>\frac{1}{2}$" while the second observation is "Head".

If I want to update $p$ using Baye's rule, it will be a lot easier if I can process the second event first and then the first event later as Beta is a conjugate prior of binomial experiments.

However, if I have to update $p$ in the order of the events (first observation first, and then the second one later), the process requires a bit more of computation.

So, my question is that does the order of events matter in Bayesian updating? If not, what can be a theoretical background that justifies it?

$\endgroup$
4
$\begingroup$

AFAIK, you cannot say that $p > \frac{1}{2}$ or even $\mathbb E[p]>\frac{1}{2}$ is an "observation" or "event", but rather a constraint on your model parameter(s). The term "observation" is usually reserved for specific realizations of a random variable (i.e. draws from a distribution). In your model, one could plausibly observe $p$ (numbers between 0 and 1) or the outcomes of $Bernoulli(p)$ (either 0 or 1). There is no way to observe $\mathbb E[p]>\frac{1}{2}$, this information lies outside your probabilistic model.

As a rough rule (some caveats apply) you should be able to simulate observations from your probabilistic model, given model parameters. How would you simulate a model where $\mathbb E[p]>\frac{1}{2}$ is a possible observation?

If you start with the $p\sim Beta(a_0,b_0); a_0, b_0 \in \mathbb{R}^+$ model and then learn that $\mathbb E[p]>\frac{1}{2}$, it means your initial model was incorrect and you should change your model to reflect the constraint ($\mathbb E[p]>\frac{1}{2}$ implies $\frac{a_0}{a_0 + b_0} > \frac{1}{2}$, so some combinations of $a_0$ and $b_0$ are ruled out). Adding a constraint cannot be AFAIK directly handled in the language of Bayesian updating.

Hope that helps.

$\endgroup$
  • $\begingroup$ Thank you for your comment, @Martin Modrak. I got your point. I modified the question. What happens if the first obvservation is $\mathbb E[p]>\frac{1}{2}?$ Within the model with the Beta conjugate, there are infinitely many possibilities that can draw $\mathbb E[p]>\frac{1}{2}$. In that case, are the two events interchangable? $\endgroup$ – Andeanlll Sep 4 at 2:55
  • $\begingroup$ @Andeanlll I don't think $\mathbb E[p]>\frac{1}{2}$ can be treated as an observation either. I tried to expand my answer on that. $\endgroup$ – Martin Modrák Sep 4 at 5:19
3
$\begingroup$

In order for this to be the case, the random variables must be exchangeable.

Your example is a little different since $p>1/2$ isn't an event. An event should be in the support of the likelihood. In this case, events are only constituted by binomial random variables, or sums thereof.

$\endgroup$
2
$\begingroup$

In Bayesian inference, terms like "observation" and "event" are just conveniences; there is no fundamental importance to them, so don't get hung up on them.

In particular, there is no physical causality or time's arrow -- no "events". Whether you can carry out some calculations in more than one order depends solely on the form of the model. If, algebraically, the results are the same assuming different orders of assignments to some variables (i.e., "observations"), then, terrific, you can do whatever is convenient. If not, well, so what?

About the representation of p > 1/2, you could represent that as a likelihood function which is just a step at 1/2. That is, it is zero to the left of 1/2, and any positive constant to the right. Note that ordinary "observations" yield likelihood functions which vary smoothly, but the smoothness is not a requirement.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.