When I look at the definition of p value carefully:

$$ p = Pr(X<x|H_0) $$

where $H_0$ means the null hypothesis is true, thus condition negative. That having a test statistic $X<x$ means rejecting the hypotheiss, so I thought it matches the definition of false positive rate (FPR) based on the confusion table from Wiki, which is 1 - specificity.

Therefore, p-value + specificity = 1, and controlling p-value below a certain cutoff (i.e. $\alpha$) is equivalent to controlling specificity above 1 - $\alpha$. Is such reasoning correct? I feel uncertain because I haven't heard people discussing p-value together with specificity a lot.

Update (2018-05-23):

As pointed out by @Elvis, I made a conceptual mistake. It should've been

$$\alpha + \textsf{specificity} = 1$$

instead of p-value. Note that p-value is a random variable, while $\alpha$ and specificity are constants.

In other words, when we conduct a null-hypothesis test, we enforce the specificity of the test to be 1 - $\alpha$. This idea becomes obvious with a thought experiment.

To match the condition null-hypothesis being true, you take two samples from the same distribution, then you conduct a t-test. You do it for N times. If you set $\alpha=0.05$, then 5% of the times you would end up with a p-value < 0.05, thus rejecting the corresponding hypotheses. Since during each test, the samples are always drawn from the same distribution, so the null hypotheses are always true, and the rejected hypotheses result in type I errors. Therefore, the specificity is 0.95, the FPR is 0.05.

On a separate note brought up by @Tim, besides illustrating the sum of $\alpha$ and specificity being one, the above also demonstrates another problem that is p-value tells you nothing about FDR. In the above thought experiment, the false discovery rate (FDR) is 1, i.e. rejected hypotheses are all false positives/discoveries, because the prior probability is 0 (always null) regardless of the specificity. This issue has been discussed extensively. Besides @Tim's mention of a few posts, I recommend

The first one is very readable. The second one exposes the seriousness of this problem. The third one tries to be more optimistic, but I thought the conclusion is still serious. Besides, I alos plotted the FDR as a function of sensitivity and specificty under different prior probabilities ($r$). The key insight is that When the prior probability is low, even when the specificity and sensitivity (less important) are high, the FDR could still be very high. For example, when the prior probability is 0.01, the specificity is 0.95 (corresponding to $\alpha = 0.05$ as commonly used) and the sensitivity is 0.8 (aka. power), the FDR is still as high as FDR. However, if $\alpha$ is set to 0.001, you would see a sharp drop in FDR as shown below (1st subplot). In general, $\alpha = 0.05$ is often too generous, which leads to much frustration in replicability and even abandoning p-values.

enter image description here

The plotted function is

$$\mathsf{FDR} = \frac{N (1 - r) (1 - \mathsf{specificity})}{N(1 - r)(1 - \mathsf{specificity}) + Nr\mathsf{Sensitivity}}$$

Details about plotting are provided in this Jupyter notebook.

  • 1
    $\begingroup$ Are you sure you're not confusing $P(X|H_0)$ with $P(H_0|X)$? $\endgroup$ – Tim May 22 '18 at 21:45
  • 1
    $\begingroup$ What do you mean? I am sure p value is conditioned on null hypothesis being True $\endgroup$ – zyxue May 22 '18 at 21:56
  • 2
    $\begingroup$ Looking at your comments and edits, you seem to want to discuss your observations and ideas. Please notice that this is (just) a Q&A site, not a discussion forum, so it's not the best place for discussion. Could you please make it clear what exactly is your question (as you already got upvoted answers)? If the question provoked other questions, it'd be better to start new questions rather then let the question evolve (again, it's Q&A site). $\endgroup$ – Tim May 23 '18 at 17:34

There's some confusion between $\alpha$, the type I error rate, and the $p$-value. I'll try to explain these two concepts before turning to specificity.

You consider a test statistic $X$, with left deviations of $X$ going against the null hypothesis $H_0$ (this is a bit unusual, in general right deviations go againt $H_0$ but it doesn't matter).

Given some $\alpha \in (0,1)$, usually $\alpha = 0.05$, there's some threshold $a$ such that the test procedure $$ \text{reject } H_0 \text{ if } X < a $$ has type I error $\alpha$. The relation between $a$ and $\alpha$ is $P_{H_0}(X < a) = \alpha$.

An equivalent test procedure can be obtained through a $p$-value: given an observed value $x_{obs}$ of the test statistic, the $p$-value is $$ p = P_{H_0}(X < x_{obs}), $$ and the rejection rule is now "reject $H_0$ if $p < \alpha$".

So $\alpha$ is some known value, fixed when you design your test procedure (usually to $\alpha = 0.05$). The $p$-value $p$ depends on the observed data. It is thus a random variable (with $P_{H_0}(p < \alpha) = \alpha$).

Now what is specificity? Briefly put, it is the probability of ''not'' rejecting $H_0$, when $H_0$ is true (cf Wikipedia), that is $$ Sp = P_{H_0}(X > a) = 1 - \alpha.$$ So you have $$ \alpha + Sp = 1.$$

But you don't have $p + Sp = 1$ -- this doesn't make sense, $p$ is a random variable, $Sp$ is a constant.

  • $\begingroup$ @Elvis, so I was pretty close, am I right? I just confused p-value and $\alpha$, although a significant one, in this case. In a sense, we enforced the specificity to be 1 - $\alpha$ when conducting a hypothesis test, thus specificity is a constant, right? On a separate note, I just would like confirm that the $\alpha$ in a multiple-hypothesis setting could mean very difference things depending on how type I error is defined (e.g. familywise error rate, or false discovery rate as in Benjamini-Hochberg method, etc.), is that right? $\endgroup$ – zyxue May 23 '18 at 14:27
  • 1
    $\begingroup$ 1 - yes, the confusion between $p$ and $\alpha$ is not that uncommon. It's good to clarify this point. 2 - Specificity and Sensitivity are different ways to look at $\alpha$ and $\beta$. They are more used when the test is used as a classifier (e.h. healthy / diseased as in the Wikipedia article) but you can find them in a purely statistical context too. 3 - In the multiple testing setting, you have to make the difference between the level $\alpha^*$ of each test and the FWER $\alpha$. If you use FDR, $\alpha$ has another meaning, as you stated. $\endgroup$ – Elvis May 23 '18 at 15:00

You seem to be referring to type I and type II errors, that are among the fundamental concepts of null hypothesis testing. If $H_0$ is true, then we expect to see $\alpha$ proportion of false positives (incorrectly reject), we call this type I error. On another hand, if $H_0$ is false, we expect to see $\beta$ false negatives (fail to reject), we call this statistical power of the test.

$$ \begin{array}{cc} & \text{Not reject} & \text{Reject} \\ H_0 \,\text{is true} & \text{True negatives} & \text{Type I error} \\ H_0 \,\text{is false} & \text{Type II error} & \text{True positives} \end{array} $$

So yes, $\alpha$ controls the ratio of false positives when $H_0$ is true. However, as nicely stated in recent blog entry by Steve Luck,

. . . this is a statement about what happens when the null hypothesis is actually true. In real research, we don't know whether the null hypothesis is actually true. If we knew that, we wouldn't need any statistics! In real research, we have a p value, and we want to know whether we should accept or reject the null hypothesis. The probability of a false positive in that situation is not the same as the probability of a false positive when the null hypothesis is true. It can be way higher.

Below, I post one of his self-explanatory figures showing the point.

enter image description here

Similar discussion can be found in recent Tweet thread by F. Perry Wilson (it seems recently everyone on the Internet is discussing $p$-values). Basically, if you don't know if $H_0$ is true, or at least don't know the probability that $H_0$ is true, then $p$-value gets pretty meaningless when interpreted as a probability.

  • $\begingroup$ If $H_0$ is true, then we expect to see $1-\alpha$ proportion of false positives (incorrectly reject), we call this type I error, did you confuse $\alpha$ and $1 - \alpha$? Similarly with $\beta$ and $1 - \beta$. $\endgroup$ – zyxue May 23 '18 at 14:18
  • $\begingroup$ The second part of your answer is about "p-value telling you nothing about false discovery rate", as it highly depends on the prior probability (i.e. $p(H_1)$ in your Table 1), paraphrased from An investigation of the false discovery rate and the misinterpretation of p-values. I actually plotted the effect in a notebook, the two towards the end, they have unintuitive shapes. $\endgroup$ – zyxue May 23 '18 at 14:22
  • $\begingroup$ @zyxue right too much edits before posting, corrected the symbols. $\endgroup$ – Tim May 23 '18 at 14:26
  • $\begingroup$ @zyxue As about prior probability: yes that's exactly the problem. The prior probability is unknown, what's makes it meaningless. $\endgroup$ – Tim May 23 '18 at 14:28
  • $\begingroup$ I see your point, but I wouldn't say pointless if there is an estimate of the prior probability, or if $\alpha$ is set to be much smaller (e.g. 0.001 instead of the commonly used 0.05 (unfortunately)), then prior probability (if not too extreme) would have a much less effect based on my and the paper mentioned in my last comment. $\endgroup$ – zyxue May 23 '18 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.