Calculate fraction of multiple tests for which the null hypothesis was rejected

Question

Consider a test (e.g., a permutation test) that tests, at the individual level, if a binary event occured or not.

• A paper that I was recently reading, for example, tested whether or not a pair of two students cheated in a multiple-choice exam (plagiarism).
• They then run such a test for every pair of students, and reject the null hypothesis for $10\%$ of all pairs.
• They then concluded that $10\%$ of all pairs cheated.

Now, I am wondering if this conclusion makes sense. For each single test, there is a certain probability that the authors identified a false positive (they chose a $5\%$ significance level). Shouldn't this imply that $5\%$ of the pairs for which the null hypothesis (no cheating) was rejected should be false positives? Can I then conclude that the share of cheaters should be by $5\%$ lower.

More generally, I wonder whether it ever makes sense to calculate the fraction for which the $H_0$ is rejected. Shouldn't I, at least, account for multiple testing?

Answer 1

In this case one should definitely account for multiple testing. I think this is a case for use of FDR (False Discovery Rates), see false-discovery-rate. You should want to control that you do not accuse too many non-cheaters of cheating. See FPR (false positive rate) vs FDR (false discovery rate) or How to estimate False Discovery Rate from p-value distribution?.

I learned about this ideas first from this paper by Brad Efron.