Is the sum of p-value and specificity 1

Question

When I look at the definition of p value carefully:

$$ p = Pr(X<x|H_0) $$

where $H_0$ means the null hypothesis is true, thus condition negative. That having a test statistic $X<x$ means rejecting the hypotheiss, so I thought it matches the definition of false positive rate (FPR) based on the confusion table from Wiki, which is 1 - specificity.

Therefore, p-value + specificity = 1, and controlling p-value below a certain cutoff (i.e. $\alpha$) is equivalent to controlling specificity above 1 - $\alpha$. Is such reasoning correct? I feel uncertain because I haven't heard people discussing p-value together with specificity a lot.

Update (2018-05-23):

As pointed out by @Elvis, I made a conceptual mistake. It should've been

$$\alpha + \textsf{specificity} = 1$$

instead of p-value. Note that p-value is a random variable, while $\alpha$ and specificity are constants.

In other words, when we conduct a null-hypothesis test, we enforce the specificity of the test to be 1 - $\alpha$. This idea becomes obvious with a thought experiment.

To match the condition null-hypothesis being true, you take two samples from the same distribution, then you conduct a t-test. You do it for N times. If you set $\alpha=0.05$, then 5% of the times you would end up with a p-value < 0.05, thus rejecting the corresponding hypotheses. Since during each test, the samples are always drawn from the same distribution, so the null hypotheses are always true, and the rejected hypotheses result in type I errors. Therefore, the specificity is 0.95, the FPR is 0.05.

On a separate note brought up by @Tim, besides illustrating the sum of $\alpha$ and specificity being one, the above also demonstrates another problem that is p-value tells you nothing about FDR. In the above thought experiment, the false discovery rate (FDR) is 1, i.e. rejected hypotheses are all false positives/discoveries, because the prior probability is 0 (always null) regardless of the specificity. This issue has been discussed extensively. Besides @Tim's mention of a few posts, I recommend

• An investigation of the false discovery rate and the misinterpretation of p-values, David Colquhoun, R. Soc. open sci. 2014 1 140216; DOI: 10.1098/rsos.140216. http://rsos.royalsocietypublishing.org/content/1/3/140216
• Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. https://doi.org/10.1371/journal.pmed.0020124
• Is Most Published Research Really False? Jeffrey T. Leek and Leah R. Jager, Annual Review of Statistics and Its Application 2017 4:1, 109-122, https://www.annualreviews.org/doi/10.1146/annurev-statistics-060116-054104

The first one is very readable. The second one exposes the seriousness of this problem. The third one tries to be more optimistic, but I thought the conclusion is still serious. Besides, I alos plotted the FDR as a function of sensitivity and specificty under different prior probabilities ($r$). The key insight is that When the prior probability is low, even when the specificity and sensitivity (less important) are high, the FDR could still be very high. For example, when the prior probability is 0.01, the specificity is 0.95 (corresponding to $\alpha = 0.05$ as commonly used) and the sensitivity is 0.8 (aka. power), the FDR is still as high as FDR. However, if $\alpha$ is set to 0.001, you would see a sharp drop in FDR as shown below (1st subplot). In general, $\alpha = 0.05$ is often too generous, which leads to much frustration in replicability and even abandoning p-values.

The plotted function is

$$\mathsf{FDR} = \frac{N (1 - r) (1 - \mathsf{specificity})}{N(1 - r)(1 - \mathsf{specificity}) + Nr\mathsf{Sensitivity}}$$

Details about plotting are provided in this Jupyter notebook.

Answer 1

There's some confusion between $\alpha$, the type I error rate, and the $p$-value. I'll try to explain these two concepts before turning to specificity.

You consider a test statistic $X$, with left deviations of $X$ going against the null hypothesis $H_0$ (this is a bit unusual, in general right deviations go againt $H_0$ but it doesn't matter).

Given some $\alpha \in (0,1)$, usually $\alpha = 0.05$, there's some threshold $a$ such that the test procedure $$ \text{reject } H_0 \text{ if } X < a $$ has type I error $\alpha$. The relation between $a$ and $\alpha$ is $P_{H_0}(X < a) = \alpha$.

An equivalent test procedure can be obtained through a $p$-value: given an observed value $x_{obs}$ of the test statistic, the $p$-value is $$ p = P_{H_0}(X < x_{obs}), $$ and the rejection rule is now "reject $H_0$ if $p < \alpha$".

So $\alpha$ is some known value, fixed when you design your test procedure (usually to $\alpha = 0.05$). The $p$-value $p$ depends on the observed data. It is thus a random variable (with $P_{H_0}(p < \alpha) = \alpha$).

Now what is specificity? Briefly put, it is the probability of ''not'' rejecting $H_0$, when $H_0$ is true (cf Wikipedia), that is $$ Sp = P_{H_0}(X > a) = 1 - \alpha.$$ So you have $$ \alpha + Sp = 1.$$

But you don't have $p + Sp = 1$ -- this doesn't make sense, $p$ is a random variable, $Sp$ is a constant.

Answer 2

You seem to be referring to type I and type II errors, that are among the fundamental concepts of null hypothesis testing. If $H_0$ is true, then we expect to see $\alpha$ proportion of false positives (incorrectly reject), we call this type I error. On another hand, if $H_0$ is false, we expect to see $\beta$ false negatives (fail to reject), we call this statistical power of the test.

$$ \begin{array}{cc} & \text{Not reject} & \text{Reject} \\ H_0 \,\text{is true} & \text{True negatives} & \text{Type I error} \\ H_0 \,\text{is false} & \text{Type II error} & \text{True positives} \end{array} $$

So yes, $\alpha$ controls the ratio of false positives when $H_0$ is true. However, as nicely stated in recent blog entry by Steve Luck,

. . . this is a statement about what happens when the null hypothesis is actually true. In real research, we don't know whether the null hypothesis is actually true. If we knew that, we wouldn't need any statistics! In real research, we have a p value, and we want to know whether we should accept or reject the null hypothesis. The probability of a false positive in that situation is not the same as the probability of a false positive when the null hypothesis is true. It can be way higher.

Below, I post one of his self-explanatory figures showing the point.

Similar discussion can be found in recent Tweet thread by F. Perry Wilson (it seems recently everyone on the Internet is discussing $p$-values). Basically, if you don't know if $H_0$ is true, or at least don't know the probability that $H_0$ is true, then $p$-value gets pretty meaningless when interpreted as a probability.