# ROC Curve AUC for Hypothesis Testing Sensitivity (Power) vs Specificity ($1-\alpha$)

(I called this a ROC curve when I posted two years ago, but it's really the empirical CDF of the p-values.)

Let's set up a hypothesis test of $$H_0: \theta=\theta_0$$ versus $$H_1: \theta\ne\theta_0$$, and let's say that I have two techniques to assess this (say equal-variance t-test versus unequal-variance Welch test for $$\theta=\mu$$). To compare the techniques, I have run some simulations and tracked how many rejections I get at $$\alpha=0.05$$.

My group tends to like $$\alpha=0.05$$, so I think this is enough for my bosses, but it's not enough for me. Maybe one test is more powerful at $$\alpha=0.05$$ but the other test is more powerful at $$\alpha=0.01$$ or $$\alpha=0.06$$. This balance of $$\alpha$$ and power reminds me of ROC curves in binary classification. In fact, I can draw a ROC curve for how powerful a hypothesis test is at various $$\alpha$$-levels, and I can calculate the AUC.

(I am open to suggestions on better ways to calculate AUC than what I give in my R code below, but I do not want that to be the focus of this question.)

set.seed(2020)
R <- 100 # more repititions in my real work, more like 10000
alphas <- seq(0,1,0.025) # finer spacing in my real work: seq(0,1,0.001)
Ps <- rep(NA,R)
for (i in 1:R){
x <- rnorm(40,0,1)
y <- rnorm(40,0.5,1)
Ps[i] <- t.test(x,y,var.equal=T)\$p.value}
plot(alphas, ecdf(Ps)(alphas), xlab="alpha", ylab="power", main="ROC")
abline(a=0, b=1, lty=3)
auc <- sum(ecdf(Ps)(alphas))/length(alphas) # area under the curve
power_05 <- length(Ps[Ps<=0.05])/R # power at alpha=0.05
auc # 0.8856098
power_05 # 0.57


The power at any given $$\alpha$$-level makes sense to me. However, what does the AUC of a test mean? There are two issues for me.

1. Should anyone care how the test performs at, say, $$\alpha=0.9$$?

2. Does bigger AUC mean more power to reject the same way that more rejections at $$\alpha=0.05$$ means more power to reject?

Has anyone explored AUC in this way?

I can't think why anyone would care how the test performs at $$\alpha \simeq 0.9$$, per se. However, the ROC curve is monotonically increasing, so the power at $$\alpha\simeq 0.9$$ bounds the power elsewhere. In practice the bound is likely to be very weak for $$\alpha \lesssim 0.1$$ or so of actual interest.
Let's consider the average power $$\int power(\alpha) p(\alpha) d\alpha$$ since power depends on $$\alpha$$, this depends on the distribution of $$\alpha$$, $$p(\alpha)$$. Thus the AUC corresponds to the expected power for a uniform distribution over $$(0, 1)$$, $$\text{AUC} = \int power(\alpha) d\alpha.$$ So the AUC tells us something about the power averaged across $$\alpha$$ (or specificity since that is $$1-\alpha$$).