# How to model the probability of a truth claim given an arrangement of eyewitness accounts supporting specific instances of that claim?

Note: the title is adapted from my Philosophy.SE question Epistemic value of multiple eyewitness accounts: single event vs. multiple events given a fixed number of eyewitnesses?. See meta discussion here.

Intuitively speaking, multiple independent eyewitness accounts of a single event are more convincing than a single eyewitness account. For example, multiple independent eyewitness accounts of a loud explosion in a remote area (e.g. from different locations and viewpoints) are more convincing/reliable than a single account (e.g. maybe the single witness hallucinated the explosion).

A bit more formally, if we define X as some truth claim about some event, process or phenomenon in the real world, we could say that:

P(X is true | multiple eyewitness accounts) > P(X is true | a single eyewitness account)

However, what happens if we keep the number of eyewitness accounts constant and only change the number of events?

For example, let X = "alien abductions are real", and let Ei be a concrete example of an (alleged) alien abduction. X is a general claim, Ei is a claim about a very specific instance of X. It is clear that Ei entails X. Thus, which of the following probabilities is greater than the others?

• P(E1 is true | N eyewitness accounts for E1)
• P(E1 is true or E2 is true or ... or EN is true | one eyewitness account for Ei, for each i in {1, ..., N})
• P(E1 is true or E2 is true or ... or EN/2 is true | two eyewitness accounts for Ei, for each i in {1, ..., N/2})
• P(E1 is true or E2 is true or ... or EN/3 is true | three eyewitness accounts for Ei, for each i in {1, ..., N/3})
• Etc.

In other words, given N eyewitness accounts, what is the optimal distribution of eyewitnesses over specific alleged instances of X that maximizes the probability of X being true? What should be more convincing, 1000 eyewitness accounts for E1, 500 eyewitness accounts for E1 + 500 eyewitness accounts for E2, etc.?

Notice that I used eyewitness reports of abductions by aliens as an illustrative example, but the reasoning can be extended to other rare events, such as reports of miracles, paranormal phenomena, angelic encounters, Bigfoot sightings, testimonials from whistleblowers (conspiracy theories), etc.

• You might be interested in my Philosophy Stack Exchange post: Baysian argument for combining extraordinary claims Aug 27, 2022 at 20:49
• What are $E_1$ and $E_2$ and how do they relate to $X$ and $\not X$? Aug 27, 2022 at 20:54
• To come up with such a probability distribution, one would need to gather lots of data about eyewitness testimonies and whether the witnessed event was verified to be true or false. This is within the realm of statistics and may be fitting for a study, but seems well outside the scope of a Q&A (unless perhaps such studies have already been conducted). But a significant issue is with the quality of accounts, so each number of witnesses may have it's own distribution, or worse, each number of witnesses for each number of events (that make up a single claim) may have it's own distribution. Aug 27, 2022 at 21:58
• I suspect there should be enough data to some sort of meaningful distribution for $P(E \mid N \: eyewitness \: accounts \: for \: E)$, but I don't think we have too many confirmed accounts of multiple events that make up a single claim to be able to come up with even a basic distribution. Note: this would simply be trying to model the data we have. Approaching the problem on a more theoretical basis, to come up with the probabilities directly, doesn't seem viable. Aug 27, 2022 at 22:00
• You might want to look at en.wikipedia.org/wiki/Condorcet%27s_jury_theorem Aug 31, 2022 at 19:08

TL;DR

The paper Holder, Hume on Miracles: Bayesian Interpretation, Multiple Testimony, and the Existence of God deals with Bayesian updating based on witness reports from multiple events. Holder only considers the case of two events, so I will adapt his calculations to the general case.

I expected, intuitively, that (other things being equal) a single event reported by 100 witnesses has a higher posterior than 100 events reported by a single witness each. This is not what happens in general. The picture is more complicated and the answer depends on a certain inequality between the reliability of witnesses and the Bayesian prior. When there are sufficiently many reports, the witnesses are reasonably reliable and the prior is very low (as with alien abductions and miracles) the all-in-one distribution of testimonies over events is better, but with less reliable witnesses and/or higher priors this reverses. It looks like for large $$n$$ the posteriors of intermediate distributions line up in between the all-in-one and one-in-all, but I did not study this closely.

Assumptions

I assume that all events $$E_i$$ are equally probable and independent with the prior probability $$p=p(E_i)$$. Each one is reported by one or more witnesses, and the testimonies are denoted $$T_j$$. I also assume that the testimonies are independent of each other and of events they are not testimonies for.

Single event

When there is a single event $$E$$ and a single testimony $$T$$ the probability is updated according to the standard formula $$p':=p(E\mid T)=\frac1{1+\frac{p(E^c)}{p(E)}\cdot\frac{p(T\mid E^c)}{p(T\mid E)}}=\frac1{1+\frac{1-p}{p}\cdot\frac{p(T\mid E^c)}{p(T\mid E)}}.$$ Let us denote $$a:=\frac{p(E^c)}{p(E)}=\frac{1-p}{p}$$, the prior odds against the event, and $$r:=\frac{p(T\mid E^c)}{p(T\mid E)}$$ the ratio of witness's reliability (or rather unreliability): if $$r>1$$ then the witness is more likely to report the event if it did not happen than if it did. By simple calculation, $$a':=\frac{1-p'}{p'}=ar$$ are the updated odds against. So after $$n$$ updates (based on independent testimonies with identical statistical characteristics) we get $$a^{(n)}=ar^n$$ and $$p^{(n)}=\frac1{1+ar^n}.\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{(1)}$$ Note that for unreliable witnesses ($$r>1$$) this posterior decreases rather than grows with $$n$$, converging to $$0$$ when $$n\to\infty$$. This is because for $$r>1$$ their reporting actually makes the event less likely.

Multiple events

Following Holder, the probability we are looking for here is $$p(E_1\cup\dots\cup E_m\mid T_1,\dots,T_n)$$ — that at least one of the reported events happened. This is not the probability of $$X$$ conditioned on $$T_j$$, but the leftover probability (of $$X$$ without any confirming events) is the same whether we are updating based on one or multiple events. So comparing to $$p(E\mid T_1,\dots,T_n)$$ gets us what we want.

By the complement rule, and taking into account independence of $$E_i$$: $$p(E_1\cup\dots\cup E_m\mid T_1,\dots,T_n)=1-\prod_{i=1}^mp(E_i^c\mid T_1,\dots,T_n).$$ Suppose we have $$n_1$$ testimonies for $$E_1$$, $$n_2$$ for $$E_2$$, and so on, $$n=n_1+\dots+n_m$$. Assuming again that all statistical characteristics are identical, and each testimony influences the probability of its event only, we have
$$p(E_1\cup\dots\cup E_m\mid T_1,\dots,T_n) =1-\prod_{i=1}^m(1-p^{(n_i)})\\ =1-\frac{a^mr^n}{\prod_{i=1}^m(1+ar^{n_i})}.\ \ \ \ \ \ \ \ \ \ \text{(2)}$$ When $$m=1,n_1=n$$ we recover the single event formula.

Comparison

Since comparing posteriors across the full range of $$m,n,a,r$$ looks hairy I will restrict to the extreme cases, $$m=1$$, $$m=n$$ (single event reported by $$n$$ witnesses, and $$n$$ events reported by a single witness each), and consider only the case of large $$n$$, which is arguably where the results become meaningful. The posterior for the latter reduces to $$\widetilde{p}^{(n)} =1-\left(\frac{ar}{1+ar}\right)^n.\ \ \ \ \ \ \ \ \ \ \text{(3)}$$ In contrast to $$p^{(n)}$$, this posterior converges to $$1$$ for any values of $$a,r>0$$, even when the witnesses are unreliable with $$r>1$$. This already tells us that for large $$n$$ this posterior is closer to $$1$$ when $$r>1$$.

When $$r<1$$, we have from calculus that $$p^{(n)}=\frac1{1+ar^n}\sim1-ar^n$$ for large $$n$$, so which one of $$(1),\ (3)$$ is larger for large $$n$$ is determined by the direction of the inequality between the bases of the exponents, $$r$$ and $$\frac{ar}{1+ar}$$. In particular, for $$p^{(n)}$$ to dominate we need $$a(1-r)>1$$. This will be the case if the prior odds against our events are high ($$a\gg1$$, which is, presumably, the case for alien abductions and miracles), and the witnesses favor what actually happened ($$r\ll1$$).

Discussion

In the case of multiple events with single reports even anti-witnesses (with $$r>1$$), who drove the probability down to $$0$$ for a single event, will now drive it up to $$1$$. This is simply the consequence of the independence of $$E_i$$ and the fact that even anti-witnesses leave the posterior of each event positive. Looking at $$(2)$$, it seems that the all-in-one and one-in-all distributions are the optimal ones for large $$n$$ (due to exponential dichotomy), but I did not prove this rigorously. One has the maximal posterior (other things being equal), the other the minimal, and the rest line up in between. Which is which is determined by the inequality $$a(1-r)>1$$.

Independence across the board is assumed above to make the calculations tractable, and is not realistic. For example, Holder calls assuming independence of $$E_i$$ "too simplistic" because "if we know that one miracle has occurred then our reasoning to the intrinsic improbability of miracles in general is wrong, and we should instead assume that they are likely". Assuming dependence will reduce the posterior for multiple events because each will contribute less to it, so in conditions that make all-in-one distribution optimal it will remain so. However, interdependence of testimonies for the same event is also more than likely, and would reduce that posterior. How all of this balances out in the end depends on how these dependencies are quantified, which is hard for me to guess.

Another confounding factor is evidence against. For example, from testimonies that confirm the validity of natural laws (in case of miracles) or debunk alleged encounters (in case of alien abductions). I suppose some of that goes into the inscrutable prior, but the quantitative effect on the posterior is hard to assess. I suspect it is a large part of why miracles and alien abductions are not widely believed despite the posteriors approaching $$1$$ for large $$n$$.

• Why would the assumptions of independence be reasonable? Evidence for one alien abduction makes testimony for any other alien abduction more likely. And almost any reason to declare that two testimonies are about the same miracle or same alien abduction is also a reason to consider those testimonies correlated. Aug 31, 2022 at 1:23
• @MattF. I agree, see additions to the Discussion. For me, it is more about comparing one event to multiple events in idealized conditions where calculations can be carried out than getting realistic posteriors. Quantifying dependence is likely to be controversial and case sensitive, I have not seen specific suggestions in the literature. Negative evidence is another factor that more realistic estimates should take into account. Aug 31, 2022 at 9:36

You use an expression like 'P(E1 is true | N eyewitness accounts for E1)', but I assume that the question is not whether E1 is true but instead whether X is true.

Instead you might consider the probability that $$X$$ is true given the number of eyewitnesses $$N_1, N_2, etc.$$

$$P(\text{X true} | N_1, N_2, \dots, N_k)$$

This can be expressed with Bayes' rule

$$P(\text{X true} | N_1, N_2, \dots, N_k) = \frac{P(N_1, N_2, \dots, N_k | \text{X true})}{P(N_1, N_2, \dots, N_k ) } P(\text{X true})$$

This expresses a computation of posterior probability or believe (given the eyewitnesses) in terms of the prior probability or believe (without the eye witnesses).

The change of $$P(\text{X true})$$ to $$P(\text{X true} | N_1, N_2, \dots, N_k)$$ depends on the probability of the eyewitnesses given that $$X$$ is true and also given that $$X$$ is false.

You can also express it as a ratio with the probability $$P(\text{X false})$$

$$\frac{P(\text{X true} | N_1, N_2, \dots, N_k) }{P(\text{X false} | N_1, N_2, \dots, N_k) } = \frac{P(\text{X true})}{ P(\text{X false})} \cdot \frac{P(N_1, N_2, \dots, N_k | \text{X true})}{P(N_1, N_2, \dots, N_k | \text{X false})}$$

The odds for $$X$$ being false or true change depending on whether the eyewitnesses are more or less probable given $$X$$ being false or true.

The problem in cases of 'esoteric theories' like alien abductions is

1. that the previous odds are very low. "extraordinary claims require extraordinary evidence"

2. and the observations like eyewitnesses do not change it much.

• When alien abductions are true, then it is very probable to have eye witnesses for it.
• But, when alien abductions are false then it is also very probable to have eye witnesses for it.

(Because it is not true does not mean that there won't be eyewitnesses that in some way believe they saw or experienced abduction)

3. Also problematic is that these mathematical formulations don't capture the entire situation very well. Asside from the number of eyewitnesses it is also important what the quality of the eyewitnesses is.

How did people experience and witness the events? For instance during the broadcast of war of the world's in 1938 a lot of people thought that aliens were landing on earth. But, they all saw (listened to) the same event where the event itself was fake. The number of witnesses might not be very useful if the event is not useful. It also matters whether some event is contrary to other previous believes (for instance do the spaceships and aliens disobey known physical laws or not).

But, possibly for some less extreme theories it might be possible to fill in some of the terms. An example could be finding the landing place of a crashing airplane that was undetected by the radar. Then the theories to consider are not whether the plane crashed or not, but more like how it had been flying. In this case observations along with triangulation may help to reduce the distribution of theories and pinpoint the crash site.

For events like meteors and earthquakes these eyewitness accounts are gathered. (I am not sure how useful it is for meteors since most often a camera is capturing it now. In the case of earthquakes it can help to learn about the spread and impact of an earthquake)

An example:

• @dipetkov adjusted Aug 28, 2022 at 9:27
• You don't discuss P(X true) / P(X false), the prior probability that the theory is true. Or that we believe is true? I would make a handwavy claim that this term dominates, given the particular subject. I suspect that many people would put very different values on these priors for a whistleblower testimonial/earthquake vs a Bigfoot sighting/angelic encounter. This may be important to consider since Spirit Realm Investigator seems to be subtly intimating that P(X true) doesn't matter and the strength of the evidence is contained in the number of witnesses alone. Aug 28, 2022 at 9:40
• @dipetkov I don't discuss it in detail but I mention it as the first point in the list of problems. A problem with this question is that there are too many problems with it and it is difficult to know where to start and to cover all of them. I tried to be handwavy about those points as well (and let the discussion about this be for the philosophical site) and tried to focus on the question 'how to model'. The modelling of eyewitness accounts can be done and is done (but just not for stuff like alien abductions). Aug 28, 2022 at 10:09
• I agree. But you still decided to answer it nevertheless. So I think it's fair game that I point out my concerns. Aug 28, 2022 at 10:10
• I have a technical question. There is a standard Bayesian formula for updating a prior when all reports are of a single event, and we want the posterior probability of that event. Does anything change in the formula when instead of an event we have a statement that is implied by multiple events, and the reports are of those multiple events? Is there an essential difference between updating probabilities of events vs those of inferred "theoretical" statements? Let's assume for the sake that all eyewitness reports are equally credible and all events are statistically independent. Aug 28, 2022 at 22:35

There’s a trade off based on the correlations among observers, with a possible quantitative argument and a clearer qualitative case.

A quantitative approach might consider $$n$$ observers in $$k$$ correlated groups, where the correlations relate the probability of an event observed by A and B with the probability of an event observed by A times the probability of an event observed by B. Then you can answer the question numerically if you can estimate the correlations within groups and the correlations between groups.

More qualitatively, suppose many people report a UFO at a particular time, but all from one neighborhood, all wearing similar eyewear, all consuming the same news source that mentions certain topics, all subject to similar reactions from friends and family and colleagues. In that case, even a large number of reports might not be convincing; reports from a more diverse group might be more persuasive.

On this logic, should you be more persuaded that an event happened if A and B report the same event, or if they reported different events?

You may consider A and B so correlated that you find an event roughly equally likely whether one or both report it. In that case you should be more convinced an event occurred when they report separate events.

Or, you may consider A and B so different that their agreement helps overcome your background skepticism. In that case you should be more convinced an event occurred when they report the same event.

The optimal number of groups in the question will likewise depend on your evaluations of the different observers and their correlations.

In my opinion, the most important thing to consider is if there are ways in which eyewitness accounts may not capture the full possibilities of the event (or lack of it), i.e., if they are statistically biased in any way, and if so, how.

Of course, "eyewitness" can be replaced with any non-human objective measurement obtained through a measurement device or procedure, and the question remains whether the measurement is biased or distorted or noisy, and if so, in what way.

A simple example to illustrate the problem is Survivorship bias, i.e., a censored dataset:

Let's assume, for argument's sake, continuing the OP's example, that there indeed are aliens and they abduct humans. But furthermore, let's assume that their abductions are very common, and that the vast majority of the humans being abducted are never returned to Earth. To make matters worse, the aliens can analyze human relationships, and they make an effort to abduct humans with very few relationships and relatives, so as not to draw attention to these missing humans.

In this hypothetical example, there will be zero eyewitnesses of humans abducted and not returned to Earth, even though the assumption is that this is the event with vast majority. The very few cases of abductions of humans which survive to report them is miniscule, obviously in comparison with humans living on Earth which have not been abducted (as far as they're aware...)

So, what would we be able to infer from existing eyewitness accounts if we ignore Survivorship bias? Probably that the existence of alien abductions is questionable. However, if we do take take into account the censored dataset caused by Survivorship bias, we would attach non-negligible probability to alien abductions. Of course this is only in the case where we accept this preposterous hypothetical example and attach to it's existence non-zero prior probability...

The example of Survivorship bias of determining reinforcement based on location of bullet holes in the returning aircraft is most illuminating: https://en.wikipedia.org/wiki/Survivorship_bias#In_the_military