p-value and the base rate fallacy The first section of this article provides some intuition on base rate fallacy with p-values. In particular, it uses as example a cancer test. 

Suppose I am testing a hundred potential cancer medications. Only ten of these drugs actually work, but I don’t know which; I must perform experiments to find them. In these experiments, I’ll look for p<0.05 gains over a placebo, demonstrating that the drug has a significant benefit.

It then calculates a hundred hypothesis tests and concludes that 

Because the base rate of effective cancer drugs is so low – only 10% of our hundred trial drugs actually work – most of the tested drugs do not work, and we have many opportunities for false positives.

The whole argument makes sense to me but I am not sure if I entirely understand how it relates to a single hypothesis test. I.e. how does the base rate fallacy creep in a single hypothesis test? Say we have setup a hypothesis test to check if the average height differs between males and females for a specific sample we collected.
 A: The positive predictive value (PPV; the probability that a drug actually working, given that we rejected the null hypothesis that it had no effect—i.e. the probability that we made a true rejection) is sensitive to the base rate of cancer drugs that actually work.
Consider the $2\times2$ table below, where testing positive or negative corresponds to rejecting or not rejecting H$_{0}$, and the truth being positive or negative means that H$_{0}$ is false or true, respectively. Each quadrant contains the counts of the four possibilities under these conditions: the number of true positive tests, number of true negative tests, number of false positive tests, and number of false negative tests. The margins sum the rows and columns, and the sum of row margins equals the sum of column margins equals the total number of tests.

PPV is the number of true positives over the total testing positive. If you imagine that the area in each quadrant of the table is proportional to the number in each quadrant, and further, imagine that the vertical line down the center of the $2 \times2$ table represents the base rate (e.g. prevalence), then the table above shows half of tests of cancer drugs truly rejecting H$_{0}$. If the base rate is lowered (that vertical line shifts left), you can see that true positives shrink relative to false positives and therefore the PPV gets smaller (i.e. just because you rejected the null hypothesis for a drug means that you still probably made a false rejection).
By contrast, the $p$ value is the probability of observing your data, if in fact the null hypothesis is true. In the table, the null hypothesis being true is the left column, and $\alpha$ (your willingness to reject the null when the null is true) is the number of false negatives over the total truly negative (or one minus the specificity of the test).
So, if the null hypothesis is true, and the base rate is low, the $p$ value being small enough to reject, even if it is very small, means that you are probably seeing a false positive.
Suggested further reading
Altman, D. G. and Bland, J. M. (1994). Diagnostic tests 2: predictive values. British Medical Journal, 309:102.
Altman, D. G. and Bland, J. M. (1994). Diagnostic tests 1: sensitivity and specificity. British Medical Journal, 308:1552.
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8):0696–0701.
