# Statistical evaluation of a 'success rate'

A recent TV show concerned with phenomena "that science cannot explain" described the work of a researcher who examined the claim that some people had telepathic powers.

I did not note down the researcher's name.
As far as I can recall, the experiment was the following:

• a group of people (the subjects of the experiment) nominated 4 friends each
• these friends phoned them (randomly, I imagine), and the test consisted in the subjects guessing who was calling
• after 800 (or 850, can't remember) trials (not per subject, but in total - I suppose they collated everything together), a 45% success rate was observed
• the researcher concluded that the observed success rate deviated very significantly from the expected random one (they showed a page from a paper, where one could read $$p = 10^{-12})$$

Ignoring for a moment the silly way things were presented, and the subject matter itself (telepathy, which is - probably rightly - considered pseudoscience https://en.wikipedia.org/wiki/Telepathy), this made me think about how one would evaluate results like these, in general (assuming that they were generated and collected in a scientifically valid way).

What question should one ask, and how would one answer, using statistics?

The fact that 25% is the 'expected' success rate if the guess is random follows from simple probability concepts, I believe, so that's not in question (is it?).

Once that is established, how does one deal with the (presumably true) observation that these subjects guessed correctly who was calling them in 45% of the cases?

Assuming the number of trials is 800, a 45% success rate means 360 correct guesses.

It's easy to calculate the probability to observe $$0, 1, ..., 360, 361, \dotsc, 800$$ guesses, under the hypothesis that the subjects were guessing randomly ($$p=0.25$$).

Let:

$$N_{trials} = n$$
$$n_{successes} = s$$
$$p_{single success} = p_s$$

Then:

$$P(s, n, p_s) = \binom {n} {s} \cdot p_s^{s} \cdot (1-p_s)^{n-s}$$

As far as I know, one would then sum the probabilities of $$360, 361, \dotsc, 800$$ successes, in this case:

sum(dbinom(360:800, 800, 0.25))
#[1] 1.114607e-34

or:

pbinom(359, 800, 0.25, lower.tail = F)
#[1] 1.114607e-34

What conclusions would one draw from this calculation (if that is the calculation one should do at all)?

• the first is that we sum probabilities of events that have not been observed. We have not measured $$361, 362, \dotsc, 800$$ successes, but still we use their probabilities to make decisions about what the only observed one ($$360$$) means.
• the second is that we focus on the fact that the subjects guessed who was calling with a larger than random ($$45\% > 25\%$$) success rate, whereas in fact it should be almost equally surprising to observe a much smaller success rate (e.g. $$5\%$$).
It's a bit as if it would be 'normal' to find that people almost always fail to guess who is calling, whereas it's 'special' to find that they guess correctly about half of the time rather than 1/4 of the time.

To address the first point, I tried out the Bayesian approach (using some theory that I already discussed here), where the observed success rate is the fixed point, and the distribution of $$p_{success}$$ is calculated.

$$PDF(p_s) = \binom {n} {s} \cdot p_s^{s} \cdot (1-p_s)^{n-s} \cdot \frac {s \cdot(n-s)} {n \cdot p_s \cdot (1-p_s)}$$

Which, plotted, gives:

And I don't know what to make out of this, either.
OK, the density of the 'random' $$0.25$$ probability of single success is still very low, but how would I derive any quantitative conclusion from this, again assuming it makes sense at all?

Any ideas?