# p-value and the base rate fallacy

The first section of this article provides some intuition on base rate fallacy with p-values. In particular, it uses as example a cancer test.

Suppose I am testing a hundred potential cancer medications. Only ten of these drugs actually work, but I don’t know which; I must perform experiments to find them. In these experiments, I’ll look for p<0.05 gains over a placebo, demonstrating that the drug has a significant benefit.

It then calculates a hundred hypothesis tests and concludes that

Because the base rate of effective cancer drugs is so low – only 10% of our hundred trial drugs actually work – most of the tested drugs do not work, and we have many opportunities for false positives.

The whole argument makes sense to me but I am not sure if I entirely understand how it relates to a single hypothesis test. I.e. how does the base rate fallacy creep in a single hypothesis test? Say we have setup a hypothesis test to check if the average height differs between males and females for a specific sample we collected.

• I suggest retitling to something like "p-value and the base rate fallacy" – Alexis Apr 28 '14 at 16:05

The positive predictive value (PPV; the probability that a drug actually working, given that we rejected the null hypothesis that it had no effect—i.e. the probability that we made a true rejection) is sensitive to the base rate of cancer drugs that actually work.

Consider the $2\times2$ table below, where testing positive or negative corresponds to rejecting or not rejecting H$_{0}$, and the truth being positive or negative means that H$_{0}$ is false or true, respectively. Each quadrant contains the counts of the four possibilities under these conditions: the number of true positive tests, number of true negative tests, number of false positive tests, and number of false negative tests. The margins sum the rows and columns, and the sum of row margins equals the sum of column margins equals the total number of tests.

PPV is the number of true positives over the total testing positive. If you imagine that the area in each quadrant of the table is proportional to the number in each quadrant, and further, imagine that the vertical line down the center of the $2 \times2$ table represents the base rate (e.g. prevalence), then the table above shows half of tests of cancer drugs truly rejecting H$_{0}$. If the base rate is lowered (that vertical line shifts left), you can see that true positives shrink relative to false positives and therefore the PPV gets smaller (i.e. just because you rejected the null hypothesis for a drug means that you still probably made a false rejection).

By contrast, the $p$-value is the probability of observing your data, if in fact the null hypothesis is true. In the table, the null hypothesis being true is the left column, and $\alpha$ (your willingness to reject the null when the null is true) is the number of false negatives over the total truly negative (or one minus the specificity of the test).

So, if the null hypothesis is true, and the base rate is low, the $p$ value being small enough to reject, even if it is very small, means that you are probably seeing a false positive.

• See my correction to the paragraph following the table. In the case of a single hypothesis test: (1) Reject H$_{0}$ height of men equals height of women; (2) pose the questions (i) what is the prevalence of true rejections of H$_{0}$ (how could one know this? One can't: it's an assumption for which there may or may not be evidence), and (ii) how does this affect PPV?; (3) if PPV is low, feel comfortable expecting that your rejection of H$_{0}$ with a low $p$-value is likely a false positive. – Alexis Apr 28 '14 at 19:57