32
$\begingroup$

A famous aphorism by cosmologist Martin Rees(*) goes "absence of evidence is not evidence of absence". On the other hand, quoting Wikipedia:

In carefully designed scientific experiments, even null results can be evidence of absence. For instance, a hypothesis may be falsified if a vital predicted observation is not found empirically. (At this point, the underlying hypothesis may be rejected or revised and sometimes, additional ad hoc explanations may even be warranted.) Whether the scientific community will accept a null result as evidence of absence depends on many factors, including the detection power of the applied methods, the confidence of the inference, as well as confirmation bias within the community.

Therefore, for the sake of scientific progress, we end up accepting the absence of evidence as evidence of absence. This is also at the heart of two very famous analogies, namely Russell's teapot and Carl Sagan's Dragon in the garage.

My question is: how can we formally justify, based on Bayesian probability theory, that absence of evidence can legitimately be used as evidence of absence? Under which conditions is that true? (the answer is expected to depend on the specific details of the problem such as the model we assume, the information gain provided by our observations given the model, or the prior probabilities of the competing hypotheses involved).

(*) the origin of the aphorism seems to be much older, see e.g. this.

$\endgroup$
14
  • 17
    $\begingroup$ I think one source of confusion is that there are two meanings of the phrase "abesence of evidence": i) No investigation or study has taken place, there is no evidence available, ii) there are some studies and they have found no support for a certain hypothesis. Personally, I think the phrase "absence of evidence is not evidence of absence" reslates to the first case, not the second. $\endgroup$ Mar 7, 2021 at 16:26
  • $\begingroup$ There's another source of ambiguity. Within probability theory, there's a clear meaning for "evidence". However, in a discussion the sentence is likely to be understood as "absence of evidence is not a certainty of absence", which completely alters the meaning and might be what the original author (Martin Rees) meant? $\endgroup$
    – rasmodius
    Mar 8, 2021 at 14:20
  • $\begingroup$ @COOLSerdash Wouldn't the second case actually be "evidence of absence"? The phrase "absence of evidence is not evidence of absence" simply means that i) and ii) are not the same. $\endgroup$
    – gardenhead
    Mar 8, 2021 at 15:28
  • 1
    $\begingroup$ @ RBarryYoung, could you elaborate a bit more? I'm sorry but I cannot see how that relates to this problem (IMO, the so-called Black Swan problem or theory is neither a problem nor a theory, just a straightforward consequence of power-law distributions. And contrary to Taleb's views, most of the extreme events in history are not Black Swans, I suggest you look for a discussion between Taleb and Didier Sornette on YouTube) $\endgroup$
    – rasmodius
    Mar 8, 2021 at 16:40
  • 1
    $\begingroup$ @COOLSerdash There's also equivalence testing which expressly looks for evidence of no effect of a given size or larger. $\endgroup$
    – Alexis
    Mar 10, 2021 at 19:58

7 Answers 7

35
$\begingroup$

There's a difference between not looking and therefore not seeing any X, and looking and not seeing any X. The latter is 'evidence', the former is not.

So the hypothesis under test is "There is a unicorn in that field behind the hill." Alice stays where she is and doesn't look. If there is a unicorn in the field, Alice sees no unicorns. If there is no unicorn in the field, Alice sees no unicorns. P(sees no unicorn | is unicorn) = P(sees no unicorn | no unicorn) = 1. When the hypothesis makes no difference to the observation, the 'evidence' contributed by the observation to belief in the hypothesis is zero.

Bob climbs to the top of the hill and looks down on the field, and sees no unicorn. If there is a unicorn in the field, Bob would see it. If there is no unicorn in the field, Bob would see no unicorn. P(sees no unicorn | is unicorn) $\neq$ P(sees no unicorn | no unicorn). When the hypothesis being true or false changes the probability of the observation, evidence is contributed. Looking and seeing no unicorns in the field is positive evidence that there are no unicorns in the field.

We can quantify evidence using Bayesian probability.

$$P(H_1|O)={P(O|H_1)P(H_1)\over P(O)}$$

$$P(H_2|O)={P(O|H_2)P(H_2)\over P(O)}$$

where $H_1$ is "there is no unicorn in that field". $H_2$ is "there is a unicorn in that field", and $O$ is "I see no unicorn". Divide one by the other:

$${P(H_1|O)\over P(H_2|O)}={P(O|H_1)\over P(O|H_2)}{P(H_1)\over P(H_2)}$$

Take logarithms to make the multiplication additive:

$$\mathrm{log}{P(H_1|O)\over P(H_2|O)}=\mathrm{log}{P(O|H_1)\over P(O|H_2)}+\mathrm{log}{P(H_1)\over P(H_2)}$$

We interpret this as saying that the Bayesian belief in favour of $H_1$ over $H_2$ after the observation is equal to the evidence in favour of $H_1$ over $H_2$ arising from the observation plus the Bayesian belief in favour of $H_1$ over $H_2$ before the observation. The additive evidence arising from the experiment is quantified as:

$$\mathrm{log}{P(O|H_1)\over P(O|H_2)}$$

Alice, by not looking, has no evidence. $\mathrm{log}(1/1)=0$. Bob, by looking and not seeing, does. $\mathrm{log}(1/0)=\infty$, meaning absolute certainty. (Of course, if there is a 10% possibility that there is an invisible unicorn in the field, Bob's evidence is $\mathrm{log}(1/0.1)=1$, if we use base 10 logs. This expresses information using a unit called the hartley.)

Rees' dictum is based on people claiming things like that there are no unicorns in the universe based on having looked at only a tiny portion of it and having seen none. Strictly speaking, there is non-zero evidence arising from this, but it's near zero, being related to the log of the volume of space and time searched divided by the volume of the universe.

Regarding the issue of null hypothesis experiments, the issue here is that often we are not able to quantify the probability of the observation given an open alternative hypothesis. What is the probability of seeing the reaction if our current understanding is wrong and some unknown physical theory is true?

So we set $H_2$ to be a null hypothesis we intend to falsify, such that the probability of the observation given the null is very low. And we presume $H_1$ is restricted to unknown alternative theories in which the observation is reasonably probable.

$$\mathrm{log}{P(H_{alt}|O)\over P(H_{null}|O)}=\mathrm{log}{P(H_{alt}|O)\over 0.05}=\mathrm{log}(20\times P(H_{alt}|O))\approx \mathrm{log}20$$

It requires some judicious assumptions about the existence of plausible alternatives, but from a Bayesian point of view doesn't look any different to any other sort of evidence.

$\endgroup$
3
  • 12
    $\begingroup$ Distilled to a sentence: absence of evidence is evidence of absence, but the strength of that evidence depends on how strong your test is (how likely it would be to succeed if the thing did exist). $\endgroup$
    – hobbs
    Mar 9, 2021 at 5:49
  • $\begingroup$ @hobbs So, the test should be undeniably capable of capturing the evidence if it exists $\endgroup$ Mar 9, 2021 at 9:41
  • $\begingroup$ @MohammedShareefC A test has some value as long as its results are not independent of the thing you're trying to measure. But the more likely it is to capture the evidence if it exists, the more useful the test is. $\endgroup$
    – Ray
    Mar 9, 2021 at 20:40
16
$\begingroup$

Bayes says he’s wrong. If observation $O$ would provide support for theory $T$, $$ P(T|O) > P(T) $$ then failure to observe that, which I denote $\bar O$, must disfavor $T$, $$ P(T|\bar O) < P(T) $$ Note that we required no special assumptions.


To see this, note that by Bayes’ theorem, the first inequality implies $$ \frac{P(O|T)}{P(O)} > 1 $$ which leads to $$ 1 - P(\bar O|T) > 1 - P(\bar O) $$ and thus $$ \frac{P(\bar O|T)} {P(\bar O)} < 1 $$ By Bayes’ theorem, this leads to the second inequality.


In lay terms, it means that if an experiment could find evidence that supports a theory, then performing the experiment but failing to find that evidence must count as evidence against the theory.

$\endgroup$
8
  • 5
    $\begingroup$ Suppose it may or may not be raining and people may or may not be carrying umbrellas and you don't know either of these things. It is impossible to maintain both that seeing people carrying umbrellas would increase your estimate of the probability that it is raining and also that seeing people not carrying umbrellas would not decrease your estimate of the probability that it is raining. $\endgroup$ Mar 8, 2021 at 3:59
  • 3
    $\begingroup$ I added some explanation. Why the downvote? This really is the correct, canonical answer afaik $\endgroup$
    – innisfree
    Mar 8, 2021 at 4:10
  • 1
    $\begingroup$ See @Eoin's answer. i.e. lack of proper observations is no evidence. Otherwise you can prove everything by that argument. $\endgroup$ Mar 8, 2021 at 13:32
  • 1
    $\begingroup$ @ak How can one prove everything by the arguments in my answer!? It just contains a quite trivial but extremely relevant lemma $\endgroup$
    – innisfree
    Mar 8, 2021 at 13:54
  • 1
    $\begingroup$ @akostadinov Lack of any observation is some evidence of absence as long as there was a nonzero probability of making the observation by chance. It's often so little evidence as to be no evidence for all practical purposes, and it certainly isn't proof of absence, but it is a nonzero amount of evidence of absence. innisfree's argument (rather, innisfree's proof; the math is correct) does not let you prove everything. It lets you say that, all else being equal, things you haven't seen are less likely to exist than things you have seen. $\endgroup$
    – Ray
    Mar 9, 2021 at 20:52
13
$\begingroup$

There's an important point missing here, but it's not strictly speaking a statistical one.

Cosmologists can't run experiments. Absence of evidence in cosmology means there's no evidence available to us here on or near earth, observing the cosmos through instruments.

Experimental scientists have a lot more freedom to generate data. We could have an absence of evidence because no one has run the appropriate experiment yet. That isn't evidence of absence. We could also have it because the appropriate experiment was run, which should have produced evidence if the phenomenon in question was real, but it didn't. This is evidence for absence. This is the idea formalised by the more mathematical answers here, in one form or another.

$\endgroup$
8
$\begingroup$

$ {\newcommand\P[1]{\operatorname{P} \left(#1\right) }} {\newcommand\PC[2]{\P{#1 \, \middle| \, #2}}} {\newcommand\A[0]{\text{no evidence}}} {\newcommand\B[0]{\text{absence}}} {\newcommand\PA[0]{\P{\A}}} {\newcommand\PB[0]{\P{\B}}} {\newcommand\PAB[0]{\PC{\A}{\B}}} {\newcommand\PBA[0]{\PC{\B}{\A}}} $tl;dr Absence-of-evidence isn't (significant) evidence-of-absence when the odds of not finding evidence aren't (significantly) affected by absence.


Following the derivation in @fblundun's answer, this claim works when $$ \PA ~=~ \PAB \,. $$

Three notable cases:

  1. When this equality holds perfectly and unconditionally,$$ \PA ~=~ \PAB \,, $$the claim is unfalsifiable.

    For example, consider a magical pink unicorn that watches us from outside of reality. But, under no circumstance does it physically interact with us in any way: it exerts no physical effects, including gravitational effects, whatsoever. Since we can never test for the existence of this magical pink unicorn, it's unfalsifiable. It doesn't really exist nor not-exist; neither claim is meaningful.

  2. When this equality is approximately true,$$ \PA ~\approx~ \PAB \,, $$then it's the sort of scenario where people tend to say

    The absence of evidence isn't the evidence of absence.

    For example, say a historian is trying to find evidence that prehistoric people saw a particular star in the night sky. They may fail to find, say, a cave-drawing with that exact star somehow depicted. But since it seems pretty unlikely for there to have been a cave-drawing of that exact star anyway, that absence-of-evidence (a lack of cave-drawings depicting the star) isn't much evidence-of-absence (evidence that prehistoric people didn't see that star).

  3. When this equality isn't approximately true,$$ \require{cancel} \PA ~\cancel{\approx}~ \PAB \,, $$then the claim

    The absence of evidence isn't the evidence of absence.

    doesn't really work.

Basically, if you wouldn't expect to find evidence of something even if it's true, then there's no sense in saying that it's false when you don't find evidence.


Discussion: How Bayesian reasoning works.

Naively, Bayesian-logic requires us to recognize that we have some pre-existing beliefs about how likely things are. Then, as we continue to make observations, we can find evidence for/against that adjust the likelihoods up/down.

Strictly speaking, anything that makes something less probable is evidence-of-absence. For example, did someone get a perfect score on their elementary-school-spelling-test last week in your hometown? You could check your local news, and if you don't see a story reporting on it, then that's technically evidence against it (since there was some possibility of it being reported, and it not being reported excludes that line of possibilities). But since it was a really small branch of probability-space (a Bayesian tree) that got excluded, this absence-of-evidence isn't (significant) evidence-of-absence.

However, absence-of-evidence can be evidence-of-absence. For example, if you have a kid in your household who brags a lot whenever they get a perfect score on a test, but you didn't hear any bragging last week, then that would be relatively significant evidence that they didn't get a perfect score last week.

Then there's the odd-ball case: unfalsifiable scenarios. Whenever a proposition being true/false would literally have no effect on reality whatsoever, then it's unfalsifiable. Unfalsifiable claims can never be evidenced, neither for nor against, so the claim that absence-of-evidence isn't evidence-of-absence holds perfectly in those scenarios. This point may come up in discussions about religion.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for your exhaustive answer. My original question, before editing it, contained the following statement: "I think the answer depends on the specific details of the problem such as the model we assume, the information gain provided by our observations given the model, or the prior probabilities of the competing hypotheses involved". This fits very well with your answer. $\endgroup$
    – rasmodius
    Mar 8, 2021 at 13:33
3
$\begingroup$

I think that @Nat's answer is good, but understates the importance of the case where $P(\mbox{absence}) \approx P(\mbox{absence} | \mbox{no evidence})$ (or not-quite-but-almost-equivalently $P(\text{no evidence}|\text{absence}) \approx P(\text{no evidence})$).

The big problem here is that, as a general rule, it is not known to the experimenter a priori if the above equality holds, and it is therefore not safe in the general case to assume that absence of evidence is evidence of absence. Taken in context, it may be possible to interpret absence of evidence as evidence of absence, but one does not necessarily follow from the other, and one should be very very careful when attempting to interpret absence of evidence as evidence of absence.

One scenario where this happens is when evidence is hard to gather for some reason or another. For example, suppose that we would like to study the difference in educational outcomes between textbook A and textbook B. We give textbook A to class A and textbook B to class B, and compare their grades afterward. Well, unfortunately, class A and class B are potentially taught by the same instructor, who has opinions on the textbooks which ultimately affect the outcome of the experiment, xor class A and B are taught by different instructors which affects the outcome of the experiment. Also, two dozen other confounding factors with varying degrees of impact on the outcome of the experiment. Ultimately, the data show nothing because of these things, and there is an absence of evidence differentiating the outcomes from textbook A and B. It's not that we didn't look, and although maybe the data were easy to gather, any potential evidence was difficult to gather. I.e. $P(\text{no evidence}) \approx P(\text{no evidence} | \text{absence}) \approx 1$. In the Bayesian context, if evidence is impossible, then one cannot update one's prior. In the real world, if you're studying gravity in a vacuum, then you maybe don't have to worry about this, but if you're studying just about anything with human subjects, then it is sometimes difficult to know ahead of time whether your experiment even can yield good evidence. In any case, if researchers do an experiment and fail to find an effect, you should seriously contemplate $P(\text{no evidence})$ in the context of the experiment before you decide to interpret an absence of evidence as evidence of absence. As another analogy, you might consider a criminal who covers his tracks well: the detective fails to find evidence not because she didn't look for it, but because the evidence was hard to find.

Yet another scenario where this happens very frequently (and somewhat surprisingly) is when $P(\text{absence}) \approx 0$. Using the same example above, consider testing the hypothesis that $H_1 :=$ textbook A gives different educational outcomes by some chosen metric than textbook B (which is an incredibly common sort of hypothesis to test). The opposite (absence) is that textbook A and textbook B have identical outcomes, which is, for most priors, impossible (it's a single point!). Since $P(\text{absence}) \approx P(\text{absence} | \text{no evidence}) \approx 0$, then when our study fails to find an effect, we should be very careful in deciding that this presents evidence of a lack of an effect. Of course, if you're using Bayesian techniques to study the problem, then you won't "fail to find an effect", but in a post hoc Bayesian analysis of the experiment that was run (since many scientists don't do the Bayesian thing), we can't interpret absence of evidence as evidence of absence.

So, given an arbitrary study that fails to find an effect, you should not interpret this as strong evidence for a lack of an effect. Without careful consideration of the context of the experiment, you shouldn't even consider it weak evidence for a lack of an effect. I think that this is the essence of the adage: "absence of evidence is not [necessarily] evidence of absence".

$\endgroup$
1
  • $\begingroup$ although maybe the data were easy to gather, any potential evidence was difficult to gather. This is a great summary of the problem that I haven't found in the other answers yet. $\endgroup$ Mar 11, 2021 at 11:57
1
$\begingroup$

Suppose absence of evidence were not evidence of absence, i.e.

P(absence | no evidence) = P(absence)

Then by Bayes' Theorem we would have

P(absence) = P(no evidence | absence)P(absence)/P(no evidence)

Multiply both sides by P(no evidence)/P(absence) to get:

P(no evidence) = P(no evidence | absence)

which means absence doesn't decrease the probability of evidence, which is absurd.

$\endgroup$
14
  • 4
    $\begingroup$ “which means absence doesn't decrease the probability of evidence” That’s precisely the point, and it’s not absurd. Whether absence of evidence is evidence of absence depends on whether we would have expected that evidence under presence. Suppose an investigation of tea prices in China yields an absence of evidence that Bob murdered Sally. That doesn’t mean that it’s evidence of absence. $\endgroup$
    – user76284
    Mar 8, 2021 at 0:39
  • 1
    $\begingroup$ @user76284 no, learning that the tea price investigation yielded no evidence that Bob murdered Sally should indeed slightly decrease your probability that Bob murdered Sally, because it makes it less likely that Bob murdered Sally as part of a scheme to rig tea prices. To put it another way: if the investigation somehow had yielded evidence that Bob murdered Sally, that should increase your probability that Bob murdered Sally. So by the law of total probability, the investigation failing to yield such evidence must have the opposite effect. $\endgroup$
    – fblundun
    Mar 8, 2021 at 1:05
  • 2
    $\begingroup$ You misunderstood the example. The point is that the investigation revealed nothing about the murder in its conclusions. For all we know, its outcome could be more likely under the opposite hypothesis. $\endgroup$
    – user76284
    Mar 8, 2021 at 5:49
  • 3
    $\begingroup$ The derivation of the final equation is spot-on, but the conclusion seems to miss the point: the claim can be proximally true. I mean, the OP asks about under what conditions is the claim true, and the above answer shows it: the claim is (proximally) true when$$\operatorname{P}\left(\text{no evidence}\right)~ \approx ~\operatorname{P}\left(\text{no evidence} \middle| \text{absence}\right) .$$ $\endgroup$
    – Nat
    Mar 8, 2021 at 11:21
  • 2
    $\begingroup$ For example, the fact that a time-traveler hasn't come back to tell us about $X$ is evidence against $X .$ But if the probability of us not seeing a time-traveler coming back to tell us about $X$ isn't significantly affected by the probability of $X ,$ then the absence-of-evidence isn't significantly evidence-of-absence. $\endgroup$
    – Nat
    Mar 8, 2021 at 11:30
0
$\begingroup$

I believe this is a very philosophical question and I doubt Bayesian theory is applicable to it. What do we mean by a "probability" of a dragon in the garage, a teapot between Earth and Mars, or extraterrestrial life? They either exist or not, and it is not a realisation of a random variable.

To drive the idea to the extreme, let's take an example from mathematics. Let's define:

\begin{align} F &\equiv \text{"Goldbach's conjecture is false", and} \\ n &\equiv \text{"an even number that cannot be partitioned as two primes"} \end{align}

As soon as we find a single $n$ satisfying the above definition, Goldbach falls. End of discussion. Or, in "probability" terms:

$$ P(F | n) = 1 $$

It is straightforward to show that the above implies:

\begin{align} P(F) &= P(n) + P(F|\bar n) P(\bar n), ~ ~ \text{or} \\ P(F|\bar n) &= \frac {P(F) - P(n)} {1 - P(n)} \end{align}

Now, if the evidence is hard to find, the conditional and unconditional "probability" of Goldbach being wrong become identical:

$$ P(n) \rightarrow 0 \Leftrightarrow P(F|\bar n) \rightarrow P(F) $$

or, in other words, the absence of evidence doesn't affect the probability of whatever we try to prove. This is, however, conditioned on the rarity of evidence. Since we have examined the numbers up to $10^{18}$ and haven't found an $n$ to disprove Goldbach, such numbers, if exist, are likely rare. Therefore, we can keep believing Goldbach.

However, as I noted above, Goldbach's conjecture is not a random variable. It is either true or not, so talking probabilities makes no sense here. We can, at best, talk about our subjective belief in its correctness.

In general, it is hard to define when the evidence is rare. The only criterion I can imagine is to make honest and serious effort to find it, and fail. E.g. make many precise experiments which still lead to no proof.

But, as a counterexample, take the Michelson-Morley experiment: Here, we (actually the physicists) took the absence of evidence as a serious evidence of absence of the aether, eventually leading to special relativity.

So, in conclusion, I'm skeptical regarding the practical relevance of your quote. And, btw, according to Quote Investigator, it is much older than Martin Rees.

$\endgroup$
16
  • 1
    $\begingroup$ @Igor F. Thank you for your answer. I corrected the origin of the quote by referring to Quote Investigator. Regarding my question, it was epistemological rather than practical. $\endgroup$
    – rasmodius
    Mar 8, 2021 at 18:46
  • 2
    $\begingroup$ Deciding that statements which must be either true or false don't have a probability causes problems. E.g. analysis of blackjack strategy is impossible unless you can think about the probability that the card currently on top of the deck is an ace. $\endgroup$
    – fblundun
    Mar 8, 2021 at 21:08
  • 1
    $\begingroup$ @fblundun Game strategies are devised under the assumptions 1) that the game can be played multiple times 2) under known and 3) fixed rules, where 4) some component of the game is random. But we don't have the option of living multiple lives in multiple universes, where extraterrestrial life sometimes develops and sometimes not. We definitely don't have the option of the same set of axioms sometimes leading to the Goldbach's conjecture being true and sometimes false. $\endgroup$
    – Igor F.
    Mar 9, 2021 at 7:06
  • 1
    $\begingroup$ @fblundun That example is fundamentally different from the Goldbach's conjecture one: We have empirical evidence to suggest the ten digits are uniformly distributed. No proof, but you can easily show that this approximately holds for the first $10^x$ digits. So a predictive model based on that would place its bets on the billionth digit not being equal to $7$. In the case of Goldbach's conjecture, you can't make any such statement without some form of Laplace smoothing. $\endgroup$ Mar 9, 2021 at 9:55
  • 1
    $\begingroup$ @Igor F., I don't understand why repeatability is important here. You say that "They either exist or not, and it is not a realisation of a random variable." Can you give examples of the realization of random variables where the source of randomness is not our epistemological uncertainty or lack of complete information? $\endgroup$
    – rasmodius
    Mar 9, 2021 at 11:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.