Absence of evidence is not evidence of absence: What does Bayesian probability have to say about it?

Question

A famous aphorism by cosmologist Martin Rees(*) goes "absence of evidence is not evidence of absence". On the other hand, quoting Wikipedia:

In carefully designed scientific experiments, even null results can be evidence of absence. For instance, a hypothesis may be falsified if a vital predicted observation is not found empirically. (At this point, the underlying hypothesis may be rejected or revised and sometimes, additional ad hoc explanations may even be warranted.) Whether the scientific community will accept a null result as evidence of absence depends on many factors, including the detection power of the applied methods, the confidence of the inference, as well as confirmation bias within the community.

Therefore, for the sake of scientific progress, we end up accepting the absence of evidence as evidence of absence. This is also at the heart of two very famous analogies, namely Russell's teapot and Carl Sagan's Dragon in the garage.

My question is: how can we formally justify, based on Bayesian probability theory, that absence of evidence can legitimately be used as evidence of absence? Under which conditions is that true? (the answer is expected to depend on the specific details of the problem such as the model we assume, the information gain provided by our observations given the model, or the prior probabilities of the competing hypotheses involved).

(*) the origin of the aphorism seems to be much older, see e.g. this.

Answer 1

There's a difference between not looking and therefore not seeing any X, and looking and not seeing any X. The latter is 'evidence', the former is not.

So the hypothesis under test is "There is a unicorn in that field behind the hill." Alice stays where she is and doesn't look. If there is a unicorn in the field, Alice sees no unicorns. If there is no unicorn in the field, Alice sees no unicorns. P(sees no unicorn | is unicorn) = P(sees no unicorn | no unicorn) = 1. When the hypothesis makes no difference to the observation, the 'evidence' contributed by the observation to belief in the hypothesis is zero.

Bob climbs to the top of the hill and looks down on the field, and sees no unicorn. If there is a unicorn in the field, Bob would see it. If there is no unicorn in the field, Bob would see no unicorn. P(sees no unicorn | is unicorn) $\neq$ P(sees no unicorn | no unicorn). When the hypothesis being true or false changes the probability of the observation, evidence is contributed. Looking and seeing no unicorns in the field is positive evidence that there are no unicorns in the field.

We can quantify evidence using Bayesian probability.

$$P(H_1|O)={P(O|H_1)P(H_1)\over P(O)}$$

$$P(H_2|O)={P(O|H_2)P(H_2)\over P(O)}$$

where $H_1$ is "there is no unicorn in that field". $H_2$ is "there is a unicorn in that field", and $O$ is "I see no unicorn". Divide one by the other:

$${P(H_1|O)\over P(H_2|O)}={P(O|H_1)\over P(O|H_2)}{P(H_1)\over P(H_2)}$$

Take logarithms to make the multiplication additive:

$$\mathrm{log}{P(H_1|O)\over P(H_2|O)}=\mathrm{log}{P(O|H_1)\over P(O|H_2)}+\mathrm{log}{P(H_1)\over P(H_2)}$$

We interpret this as saying that the Bayesian belief in favour of $H_1$ over $H_2$ after the observation is equal to the evidence in favour of $H_1$ over $H_2$ arising from the observation plus the Bayesian belief in favour of $H_1$ over $H_2$ before the observation. The additive evidence arising from the experiment is quantified as:

$$\mathrm{log}{P(O|H_1)\over P(O|H_2)}$$

Alice, by not looking, has no evidence. $\mathrm{log}(1/1)=0$. Bob, by looking and not seeing, does. $\mathrm{log}(1/0)=\infty$, meaning absolute certainty. (Of course, if there is a 10% possibility that there is an invisible unicorn in the field, Bob's evidence is $\mathrm{log}(1/0.1)=1$, if we use base 10 logs. This expresses information using a unit called the hartley.)

Rees' dictum is based on people claiming things like that there are no unicorns in the universe based on having looked at only a tiny portion of it and having seen none. Strictly speaking, there is non-zero evidence arising from this, but it's near zero, being related to the log of the volume of space and time searched divided by the volume of the universe.

Regarding the issue of null hypothesis experiments, the issue here is that often we are not able to quantify the probability of the observation given an open alternative hypothesis. What is the probability of seeing the reaction if our current understanding is wrong and some unknown physical theory is true?

So we set $H_2$ to be a null hypothesis we intend to falsify, such that the probability of the observation given the null is very low. And we presume $H_1$ is restricted to unknown alternative theories in which the observation is reasonably probable.

$$\mathrm{log}{P(H_{alt}|O)\over P(H_{null}|O)}=\mathrm{log}{P(H_{alt}|O)\over 0.05}=\mathrm{log}(20\times P(H_{alt}|O))\approx \mathrm{log}20$$

It requires some judicious assumptions about the existence of plausible alternatives, but from a Bayesian point of view doesn't look any different to any other sort of evidence.

Answer 2

Bayes says he’s wrong. If observation $O$ would provide support for theory $T$, $$ P(T|O) > P(T) $$ then failure to observe that, which I denote $\bar O$, must disfavor $T$, $$ P(T|\bar O) < P(T) $$ Note that we required no special assumptions.

---

To see this, note that by Bayes’ theorem, the first inequality implies $$ \frac{P(O|T)}{P(O)} > 1 $$ which leads to $$ 1 - P(\bar O|T) > 1 - P(\bar O) $$ and thus $$ \frac{P(\bar O|T)} {P(\bar O)} < 1 $$ By Bayes’ theorem, this leads to the second inequality.

---

In lay terms, it means that if an experiment could find evidence that supports a theory, then performing the experiment but failing to find that evidence must count as evidence against the theory.

Answer 3

There's an important point missing here, but it's not strictly speaking a statistical one.

Cosmologists can't run experiments. Absence of evidence in cosmology means there's no evidence available to us here on or near earth, observing the cosmos through instruments.

Experimental scientists have a lot more freedom to generate data. We could have an absence of evidence because no one has run the appropriate experiment yet. That isn't evidence of absence. We could also have it because the appropriate experiment was run, which should have produced evidence if the phenomenon in question was real, but it didn't. This is evidence for absence. This is the idea formalised by the more mathematical answers here, in one form or another.

Answer 4

$ {\newcommand\P[1]{\operatorname{P} \left(#1\right) }} {\newcommand\PC[2]{\P{#1 \, \middle| \, #2}}} {\newcommand\A[0]{\text{no evidence}}} {\newcommand\B[0]{\text{absence}}} {\newcommand\PA[0]{\P{\A}}} {\newcommand\PB[0]{\P{\B}}} {\newcommand\PAB[0]{\PC{\A}{\B}}} {\newcommand\PBA[0]{\PC{\B}{\A}}} $tl;dr– Absence-of-evidence isn't (significant) evidence-of-absence when the odds of not finding evidence aren't (significantly) affected by absence.

---

Following the derivation in @fblundun's answer, this claim works when $$ \PA ~=~ \PAB \,. $$

Three notable cases:

1. When this equality holds perfectly and unconditionally,$$ \PA ~=~ \PAB \,, $$the claim is unfalsifiable.

   For example, consider a magical pink unicorn that watches us from outside of reality. But, under no circumstance does it physically interact with us in any way: it exerts no physical effects, including gravitational effects, whatsoever. Since we can never test for the existence of this magical pink unicorn, it's unfalsifiable. It doesn't really exist nor not-exist; neither claim is meaningful.

2. When this equality is approximately true,$$ \PA ~\approx~ \PAB \,, $$then it's the sort of scenario where people tend to say

   The absence of evidence isn't the evidence of absence.

   For example, say a historian is trying to find evidence that prehistoric people saw a particular star in the night sky. They may fail to find, say, a cave-drawing with that exact star somehow depicted. But since it seems pretty unlikely for there to have been a cave-drawing of that exact star anyway, that absence-of-evidence (a lack of cave-drawings depicting the star) isn't much evidence-of-absence (evidence that prehistoric people didn't see that star).

3. When this equality isn't approximately true,$$ \require{cancel} \PA ~\cancel{\approx}~ \PAB \,, $$then the claim

   The absence of evidence isn't the evidence of absence.

   doesn't really work.

Basically, if you wouldn't expect to find evidence of something even if it's true, then there's no sense in saying that it's false when you don't find evidence.

---

Discussion: How Bayesian reasoning works.

Naively, Bayesian-logic requires us to recognize that we have some pre-existing beliefs about how likely things are. Then, as we continue to make observations, we can find evidence for/against that adjust the likelihoods up/down.

Strictly speaking, anything that makes something less probable is evidence-of-absence. For example, did someone get a perfect score on their elementary-school-spelling-test last week in your hometown? You could check your local news, and if you don't see a story reporting on it, then that's technically evidence against it (since there was some possibility of it being reported, and it not being reported excludes that line of possibilities). But since it was a really small branch of probability-space (a Bayesian tree) that got excluded, this absence-of-evidence isn't (significant) evidence-of-absence.

However, absence-of-evidence can be evidence-of-absence. For example, if you have a kid in your household who brags a lot whenever they get a perfect score on a test, but you didn't hear any bragging last week, then that would be relatively significant evidence that they didn't get a perfect score last week.

Then there's the odd-ball case: unfalsifiable scenarios. Whenever a proposition being true/false would literally have no effect on reality whatsoever, then it's unfalsifiable. Unfalsifiable claims can never be evidenced, neither for nor against, so the claim that absence-of-evidence isn't evidence-of-absence holds perfectly in those scenarios. This point may come up in discussions about religion.

Answer 5

I think that @Nat's answer is good, but understates the importance of the case where $P(\mbox{absence}) \approx P(\mbox{absence} | \mbox{no evidence})$ (or not-quite-but-almost-equivalently $P(\text{no evidence}|\text{absence}) \approx P(\text{no evidence})$).

The big problem here is that, as a general rule, it is not known to the experimenter a priori if the above equality holds, and it is therefore not safe in the general case to assume that absence of evidence is evidence of absence. Taken in context, it may be possible to interpret absence of evidence as evidence of absence, but one does not necessarily follow from the other, and one should be very very careful when attempting to interpret absence of evidence as evidence of absence.

One scenario where this happens is when evidence is hard to gather for some reason or another. For example, suppose that we would like to study the difference in educational outcomes between textbook A and textbook B. We give textbook A to class A and textbook B to class B, and compare their grades afterward. Well, unfortunately, class A and class B are potentially taught by the same instructor, who has opinions on the textbooks which ultimately affect the outcome of the experiment, xor class A and B are taught by different instructors which affects the outcome of the experiment. Also, two dozen other confounding factors with varying degrees of impact on the outcome of the experiment. Ultimately, the data show nothing because of these things, and there is an absence of evidence differentiating the outcomes from textbook A and B. It's not that we didn't look, and although maybe the data were easy to gather, any potential evidence was difficult to gather. I.e. $P(\text{no evidence}) \approx P(\text{no evidence} | \text{absence}) \approx 1$. In the Bayesian context, if evidence is impossible, then one cannot update one's prior. In the real world, if you're studying gravity in a vacuum, then you maybe don't have to worry about this, but if you're studying just about anything with human subjects, then it is sometimes difficult to know ahead of time whether your experiment even can yield good evidence. In any case, if researchers do an experiment and fail to find an effect, you should seriously contemplate $P(\text{no evidence})$ in the context of the experiment before you decide to interpret an absence of evidence as evidence of absence. As another analogy, you might consider a criminal who covers his tracks well: the detective fails to find evidence not because she didn't look for it, but because the evidence was hard to find.

Yet another scenario where this happens very frequently (and somewhat surprisingly) is when $P(\text{absence}) \approx 0$. Using the same example above, consider testing the hypothesis that $H_1 :=$ textbook A gives different educational outcomes by some chosen metric than textbook B (which is an incredibly common sort of hypothesis to test). The opposite (absence) is that textbook A and textbook B have identical outcomes, which is, for most priors, impossible (it's a single point!). Since $P(\text{absence}) \approx P(\text{absence} | \text{no evidence}) \approx 0$, then when our study fails to find an effect, we should be very careful in deciding that this presents evidence of a lack of an effect. Of course, if you're using Bayesian techniques to study the problem, then you won't "fail to find an effect", but in a post hoc Bayesian analysis of the experiment that was run (since many scientists don't do the Bayesian thing), we can't interpret absence of evidence as evidence of absence.

So, given an arbitrary study that fails to find an effect, you should not interpret this as strong evidence for a lack of an effect. Without careful consideration of the context of the experiment, you shouldn't even consider it weak evidence for a lack of an effect. I think that this is the essence of the adage: "absence of evidence is not [necessarily] evidence of absence".

Answer 6

Suppose absence of evidence were not evidence of absence, i.e.

P(absence | no evidence) = P(absence)

Then by Bayes' Theorem we would have

P(absence) = P(no evidence | absence)P(absence)/P(no evidence)

Multiply both sides by P(no evidence)/P(absence) to get:

P(no evidence) = P(no evidence | absence)

which means absence doesn't decrease the probability of evidence, which is absurd.

Answer 7

I believe this is a very philosophical question and I doubt Bayesian theory is applicable to it. What do we mean by a "probability" of a dragon in the garage, a teapot between Earth and Mars, or extraterrestrial life? They either exist or not, and it is not a realisation of a random variable.

To drive the idea to the extreme, let's take an example from mathematics. Let's define:

\begin{align} F &\equiv \text{"Goldbach's conjecture is false", and} \\ n &\equiv \text{"an even number that cannot be partitioned as two primes"} \end{align}

As soon as we find a single $n$ satisfying the above definition, Goldbach falls. End of discussion. Or, in "probability" terms:

$$ P(F | n) = 1 $$

It is straightforward to show that the above implies:

\begin{align} P(F) &= P(n) + P(F|\bar n) P(\bar n), ~ ~ \text{or} \\ P(F|\bar n) &= \frac {P(F) - P(n)} {1 - P(n)} \end{align}

Now, if the evidence is hard to find, the conditional and unconditional "probability" of Goldbach being wrong become identical:

$$ P(n) \rightarrow 0 \Leftrightarrow P(F|\bar n) \rightarrow P(F) $$

or, in other words, the absence of evidence doesn't affect the probability of whatever we try to prove. This is, however, conditioned on the rarity of evidence. Since we have examined the numbers up to $10^{18}$ and haven't found an $n$ to disprove Goldbach, such numbers, if exist, are likely rare. Therefore, we can keep believing Goldbach.

However, as I noted above, Goldbach's conjecture is not a random variable. It is either true or not, so talking probabilities makes no sense here. We can, at best, talk about our subjective belief in its correctness.

In general, it is hard to define when the evidence is rare. The only criterion I can imagine is to make honest and serious effort to find it, and fail. E.g. make many precise experiments which still lead to no proof.

But, as a counterexample, take the Michelson-Morley experiment: Here, we (actually the physicists) took the absence of evidence as a serious evidence of absence of the aether, eventually leading to special relativity.

So, in conclusion, I'm skeptical regarding the practical relevance of your quote. And, btw, according to Quote Investigator, it is much older than Martin Rees.