(I called this a ROC curve when I posted two years ago, but it's really the empirical CDF of the p-values.)

Let's set up a hypothesis test of $H_0: \theta=\theta_0$ versus $H_1: \theta\ne\theta_0$, and let's say that I have two techniques to assess this (say equal-variance t-test versus unequal-variance Welch test for $\theta=\mu$). To compare the techniques, I have run some simulations and tracked how many rejections I get at $\alpha=0.05$.

My group tends to like $\alpha=0.05$, so I think this is enough for my bosses, but it's not enough for me. Maybe one test is more powerful at $\alpha=0.05$ but the other test is more powerful at $\alpha=0.01$ or $\alpha=0.06$. This balance of $\alpha$ and power reminds me of ROC curves in binary classification. In fact, I can draw a ROC curve for how powerful a hypothesis test is at various $\alpha$-levels, and I can calculate the AUC.

(I am open to suggestions on better ways to calculate AUC than what I give in my R code below, but I do not want that to be the focus of this question.)

R <- 100 # more repititions in my real work, more like 10000
alphas <- seq(0,1,0.025) # finer spacing in my real work: seq(0,1,0.001)
Ps <- rep(NA,R)
for (i in 1:R){
    x <- rnorm(40,0,1)
    y <- rnorm(40,0.5,1)
    Ps[i] <- t.test(x,y,var.equal=T)$p.value}
plot(alphas, ecdf(Ps)(alphas), xlab="alpha", ylab="power", main="ROC")
abline(a=0, b=1, lty=3)
auc <- sum(ecdf(Ps)(alphas))/length(alphas) # area under the curve
power_05 <- length(Ps[Ps<=0.05])/R # power at alpha=0.05
auc # 0.8856098
power_05 # 0.57

Plot of ROC

The power at any given $\alpha$-level makes sense to me. However, what does the AUC of a test mean? There are two issues for me.

  1. Should anyone care how the test performs at, say, $\alpha=0.9$?

  2. Does bigger AUC mean more power to reject the same way that more rejections at $\alpha=0.05$ means more power to reject?

Has anyone explored AUC in this way?


1 Answer 1


I can't think why anyone would care how the test performs at $\alpha \simeq 0.9$, per se. However, the ROC curve is monotonically increasing, so the power at $\alpha\simeq 0.9$ bounds the power elsewhere. In practice the bound is likely to be very weak for $\alpha \lesssim 0.1$ or so of actual interest.

Let's consider the average power $$ \int power(\alpha) p(\alpha) d\alpha $$ since power depends on $\alpha$, this depends on the distribution of $\alpha$, $p(\alpha)$. Thus the AUC corresponds to the expected power for a uniform distribution over $(0, 1)$, $$ \text{AUC} = \int power(\alpha) d\alpha. $$ So the AUC tells us something about the power averaged across $\alpha$ (or specificity since that is $1-\alpha$).

In other words, in answer to your second question, greater AUC tells us a test has more power when averaged uniformly over specificity.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.