Define $X:=$ "coin has probability 1 to land heads" Assume that one has the prior belief: $P(X)= 1$. However after tossing the coin once it lands tails ($E:= $ "coin landed tails"). How should a Bayesian update his beliefs in order to stay coherent? $P(X|E)$ is undefined, as $P(E) = 0 $. However, it seems to me that as his prior beliefs are quite implausible (of course probability 0 does not mean impossible) he should somehow be able to update his belief according to some rule.

Is this just a pathological case in which Bayesian updating does not work or am I unaware of a solution to this problem?

  • 13
    $\begingroup$ I think this question is far broader in scope than Bayesian analysis. Isn't it really asking what one should do in face of evidence that one's assumptions are incorrect? I would hesitate to call these situations "pathological" because they happen all the time. What would truly be pathological are situations where people refuse to change their assumptions (or beliefs) in the face of incontrovertible evidence. (Such people are usually called "politicians" rather than "Bayesians" :-).) $\endgroup$
    – whuber
    Mar 15, 2018 at 14:05
  • 2
    $\begingroup$ @whuber I am all for poking fun and scorn at (the wrong kind of) politicians, but science is not immune either. Planck remarked in his Autobiography that a new theory sometimes only triumphs when the older generation who refused to take it seriously have all died off. $\endgroup$
    – Nick Cox
    Mar 15, 2018 at 14:31
  • $\begingroup$ @Nick I'm sure you understand the situation in science is more complex than that. (Yes, the situation in politics is more complex too... .) Half a century ago, Thomas Kuhn was among the first who appreciated that and elucidated the deeper reasons. $\endgroup$
    – whuber
    Mar 15, 2018 at 14:38
  • 2
    $\begingroup$ @whuber Agreed. Good scientists change their minds quickly in the face of logic and evidence, and many of us throw away many lousy ideas before even trying to go public with them. (Minute detail: It was Kuhn's most famous book where I think I first encountered the Planck reference.) $\endgroup$
    – Nick Cox
    Mar 15, 2018 at 14:42
  • $\begingroup$ Why "Assume that one has the prior belief: $P(X)=1"? That seems to be the cause of any difficulty. An event with probability 0 did not occur (by definition!) even if an ill-formed prior was proposed under which it did. A good Bayesian, like a good scientist, will update the model on the basis of new evidence when it is obviously necessary. $\endgroup$ Feb 25 at 2:59

3 Answers 3


Any posterior probability is valid in this case

This is an interesting question, which gets into the territory of the foundations of probability. There are a few possible approaches here, but for reasons that I will elaborate on soon, the approach I favour is to give a broader definition of conditional probability that is analogous to its definition when dealing with continuous random variables. (Details of this method are shown below.) In this particular case, this leads to the conclusion that the Bayesian can hold any posterior belief about $X$, and this yields a coherent set of beliefs (notwithstanding that they have observed an event that they believe to have probability zero).

The advantage of this approach is that it gives a well-defined posterior distribution, and allows the Bayesian to update their beliefs conditional on observing an event that was stipulated to occur with probability zero. The posterior is updated essentially arbitrarily (any posterior probability is equally coherent), but that flexibility is unsurprising given what has occurred. In this case, different Bayesians with the same prior beliefs could legitimately come to different posterior conclusions, owing to the fact that they have all observed an event with zero probability a priori. This outcome is a "feature", not a "bug", since it allows the Bayesian to "escape" their screw-up in formulation of an initial model that led to a bad prediction.

Conditional probability for continuous random variables: When we are dealing with continuous random variables, the conditional probability function is defined through the Radon-Nikodym derivative, and essentially just requires the function to satisfies the law of joint probability. If $X$ and $E$ were continuous random variables (rather than discrete events) in a probability space $(\Omega, \mathscr{G}, P)$ then we would define the conditional probability function $p(x|e)$ as any non-negative measureable function that satisfies the integral equation:

$$p(x) = \int \limits_\mathscr{E} p(x|e) \ dP(e) \quad \quad \quad \text{for all } x \in \mathscr{X} \in \mathscr{G}.$$

Since $p(x)$ is also defined via the Radon-Nikodym derivative, this implicitly means that $p(x|e)$ can be any non-negative measureable function that satisfies the integral equation:

$$\mathbb{P}(X \in \mathcal{A}) = \int \limits_\mathcal{A} \int \limits_\mathscr{E} p(x|e) \ dP(e) \ dx \quad \quad \quad \text{for all } \mathcal{A} \in \mathscr{G}.$$

This gives a non-unique solution for the conditional probability function, though in practice, every solution is "almost surely" equivalent (i.e., they differ only on a set of outcomes with probability zero) so there is no problem with the non-uniqueness.

Defining conditional probability for discrete events: The standard definition for conditional probability for discrete events is the well-known ratio formula, where the denominator is the probability of the conditioning event. Obviously, in the case where the conditioning event has zero probability, this object is undefined. The obvious solution here is to broaden the definition in a manner that is analogous to the method used in the continuous case. That is, we define the conditional probability pair $\mathbb{P}(X|E)$ and $\mathbb{P}(X|\bar{E})$ as any pair of values between zero and one that satisfy the equation:

$$\mathbb{P}(X) = \mathbb{P}(X|E) \times \mathbb{P}(E) + \mathbb{P}(X|\bar{E}) \times (1-\mathbb{P}(E)).$$

In the case stipulated in the question we have the prior belief $\mathbb{P}(X) = 1$ and the sampling distribution $\mathbb{P}(E|X) = 0$, which leads to $\mathbb{P}(E) = 0$. Substituting these values into the above equation gives:

$$1 = \mathbb{P}(X|E) \times 0 + \mathbb{P}(X|\bar{E}) \times 1.$$

We can see that this equation is satisfied by taking $\mathbb{P}(X|\bar{E}) = 1$ and any $0 \leqslant \mathbb{P}(X|E) \leqslant 1$. Thus, the (posterior) conditional probability $\mathbb{P}(X|E)$ may coherently be any value between zero and one. When we say that this is "coherent" we simply mean that the posterior probability is not inconsistent with the other stipulated probabilities in the problem (i.e., the prior and sampling probabilities).

Why this approach makes the most sense: It is entirely possible that a Bayesian analysis could involve observation of a discrete event that has zero probability stipulated in the prior distribution. For example, in a standard model of coin-flipping, we stipulate a Bernoulli distribution for the heads/tails outcome, but it is possible that the coin could come to rest on its edge (thus being neither heads or tails). Brains should not explode in this case, and thus it is incumbent on Bayesian reasoning to have a well-defined way of proceeding in this case.

The major advantage of the approach I have outlined is that it always leads to at least one allowable value for the posterior probability (i.e., the posterior probability is well-defined). The posterior probability is not uniquely defined, but that is a natural offshoot of the fact that there are several values that are equally coherent with the zero-probability sampling observation. This approach means that the Bayesian is free to stipulate any posterior probability, and this is as coherent as any other. (Bear in mind that when we say "coherent" here, we are talking about coherence with a prior belief that stipulated zero probability for a discrete event that actually happened, so coherence with that is not a high bar!)

There is another major benefit to this approach, which is that it allows the Bayesian to update his or her beliefs in response to observing an event that had zero sampling probablity under the prior, and in particular, the Bayesian can now revise his or her beliefs so that they no longer ascribe zero probability to this event. In the example you give, the Bayesian had a prior belief that $X$ is true almost surely, buy then observed an event with zero sampling probability conditional on this event. Now the Bayesian is free to update his or her belief to a posterior probabilty for $X$ that is not one (and so a corresponding posterior probability for $\bar{X}$ that is not zero). So, in essence, the Bayesian can now say "Oh shit! That was a silly prior! Let me update my belief in that event so that it no longer occurs almost surely!" Moreover, this is not some ad hoc change, but a legitimate "coherent" updating done under Bayes' theorem.


This is related to field of logic. in particular, a false statement implies all other statements, true or false. In your scenario $X$ is a false statement.This means we can write $X\implies S$ for any other proposition $S$. For example, we have $X\implies E$ (it implies tails) and also $X\implies E^c$ (it implies not tails as well)!

This is consistent with Ben's solution as well (set the posterior to any value you want). Obviously this not really useful in applications though. For I am quite confident of not needing any mathematical framework for making up whatever results I want.

What it does mean, is that one should not include known false statements into their prior probabilities. This is just the same as one shouldn't use false statements about the data. In terms of dealing with "black swan" type of issues, we can deal with this conceptually by assigning some tiny, but nonzero chance that our "working assumptions" are wrong. If you call this statement $A_w$ as "my working assumptions are correct", and set it's prior equal to $p(A_w)=1-\epsilon$. There are some impossible situations under the working assumption, which means that the likelihood $p(d\in D_{impossible}|A_w)=0$ for some values of the "data" $d$ that exist in the "impossible" region $D_{impossible}$ when the working assumptions hold. Call this event $Q:=d\in D_{impossible}$. This also means that $p(Q^c|A_w)=1-p(Q|A_w)=1$. We assume that $p(Q|A_w^c)=\delta>0$ (ie "impossible" data is possible if the working assumption is wrong). And finally that $p(Q^c|A_w^c)=1-p(Q|A_w^c)=1-\delta$.

Now we have two scenarios. The first is that the data is "normal" (meaning $Q^c$ is true)

$$p(A_w|Q^c)= \frac{p(A_w)p(Q^c|A_w)}{p(A_w)p(Q^c|A_w)+p(A_w^c)p(Q^c|A_w^c)}= \frac{1-\epsilon}{1-\epsilon+\epsilon(1-\delta)}=\frac{1-\epsilon}{1-\delta \epsilon} > 1-\epsilon$$

The second is that the data is "impossible" (meaning $Q$ is true)

$$p(A_w|Q)= \frac{p(A_w)p(Q|A_w)}{p(A_w)p(Q|A_w)+p(A_w^c)p(Q|A_w^c)}=\frac{0}{0+\epsilon\delta}=0.$$

Now hopefully this very clearly shows that if your assumptions are satisfied, and you already had a very high prior probability, the posterior is at least as high (and often higher). So whatever value for $\epsilon$ you use to represent "basically impossible" before seeing the data, you should use a value that is smaller after seeing the predictions confirmed.

When doing calculations, assuming $p(A_w|Q^c)=1$ won't lead you astray. So you "absorb" $A_w,Q^c$ into the prior information

Now, what about when the impossible thing happens? Well then you need to unpack and change your likelihood and prior according to what was wrong with your assumption.

  • 1
    $\begingroup$ This is an interesting answer (+1). I have taken the liberty of making a small correction to account for the case where $\gamma = 1$, which was not excluded in your antecedent conditions. If you intended to impose the restriction $\gamma < 1$ to obtain a strict inequality, please edit accordingly. $\endgroup$
    – Ben
    May 5, 2020 at 22:35
  • $\begingroup$ I realised that $\gamma=1-\delta$ in my original answer. As $\delta>0$ this means the inequality should be strict. $\endgroup$ May 6, 2020 at 6:35

There's an implicit assumption in all reasoning, Bayesian or otherwise, that we know everything that could happen and accounted for it. If something happens which is impossible under the model, it just means that that assumption is false. The principled thing to do is to go back and expand the model, and start over. At least in a Bayesian framework, this process is relatively easy to formalize -- instead of inference within a single model, one would do inference in a set of models.

At some point, our human ability to nest models within models must run out. Even with automated help (i.e. computers or whatever), there must be an upper limit to the complexity of the "mother of all models". I don't have any idea what to do in that circumstance, but we are certainly very far away from that, when we're working with typical parametric models found in applications.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.