ROC Curve AUC for Hypothesis Testing Sensitivity (Power) vs Specificity ($1-\alpha$)

Question

(I called this a ROC curve when I posted two years ago, but it's really the empirical CDF of the p-values.)

Let's set up a hypothesis test of $H_0: \theta=\theta_0$ versus $H_1: \theta\ne\theta_0$, and let's say that I have two techniques to assess this (say equal-variance t-test versus unequal-variance Welch test for $\theta=\mu$). To compare the techniques, I have run some simulations and tracked how many rejections I get at $\alpha=0.05$.

My group tends to like $\alpha=0.05$, so I think this is enough for my bosses, but it's not enough for me. Maybe one test is more powerful at $\alpha=0.05$ but the other test is more powerful at $\alpha=0.01$ or $\alpha=0.06$. This balance of $\alpha$ and power reminds me of ROC curves in binary classification. In fact, I can draw a ROC curve for how powerful a hypothesis test is at various $\alpha$-levels, and I can calculate the AUC.

(I am open to suggestions on better ways to calculate AUC than what I give in my R code below, but I do not want that to be the focus of this question.)

set.seed(2020)
R <- 100 # more repititions in my real work, more like 10000
alphas <- seq(0,1,0.025) # finer spacing in my real work: seq(0,1,0.001)
Ps <- rep(NA,R)
for (i in 1:R){
    x <- rnorm(40,0,1)
    y <- rnorm(40,0.5,1)
    Ps[i] <- t.test(x,y,var.equal=T)$p.value}
plot(alphas, ecdf(Ps)(alphas), xlab="alpha", ylab="power", main="ROC")
abline(a=0, b=1, lty=3)
auc <- sum(ecdf(Ps)(alphas))/length(alphas) # area under the curve
power_05 <- length(Ps[Ps<=0.05])/R # power at alpha=0.05
auc # 0.8856098
power_05 # 0.57

The power at any given $\alpha$-level makes sense to me. However, what does the AUC of a test mean? There are two issues for me.

1. Should anyone care how the test performs at, say, $\alpha=0.9$?

2. Does bigger AUC mean more power to reject the same way that more rejections at $\alpha=0.05$ means more power to reject?

Has anyone explored AUC in this way?

Answer 1

I can't think why anyone would care how the test performs at $\alpha \simeq 0.9$, per se. However, the ROC curve is monotonically increasing, so the power at $\alpha\simeq 0.9$ bounds the power elsewhere. In practice the bound is likely to be very weak for $\alpha \lesssim 0.1$ or so of actual interest.

Let's consider the average power $$ \int power(\alpha) p(\alpha) d\alpha $$ since power depends on $\alpha$, this depends on the distribution of $\alpha$, $p(\alpha)$. Thus the AUC corresponds to the expected power for a uniform distribution over $(0, 1)$, $$ \text{AUC} = \int power(\alpha) d\alpha. $$ So the AUC tells us something about the power averaged across $\alpha$ (or specificity since that is $1-\alpha$).

In other words, in answer to your second question, greater AUC tells us a test has more power when averaged uniformly over specificity.