Is it possible for a variable that was proved to be significant from the two-sample t-test to have ROC curve that is close to or below the line x=y?

Question

Say we have a large sample size for each of the two groups, so that the central limit theorem can be applied and thus t-test to compare the two groups means can be justified.

Say group mean difference of the variable A was proved to be significantly different from zero in the t-test at alpha=0.05. This means the group means of variable A are significantly different between the two groups.

Is it possible that, when I draw the ROC curve of the variable A, the ROC curve of A will be close to or below the 45-degree line x=y? I wouldn't think this will happen since if the group means of variable A are significantly different between the two groups, that means A can be a good variable to use to distinguish between the two groups.

If such phenomenon is possible to take place, what can be the reason for it?

Thank you,

Answer 1

"Statistically significant" doesn't mean that the difference between the group is large, nor that is it relevant in any meaningful way. It only means that you can rule out that it happened by chance. This is especially true when you have a large number of observations: even tiny differences in mean can become very significant.

Let's take an example. Two samples of 10 000 000 randomly distributed observations, with a small difference in means:

a <- rnorm(10000000)
b <- rnorm(10000000) + 0.001

Because of the sheer sample size, that's going to be very significantly different:

t.test(a, b)

    Welch Two Sample t-test

data:  a and b
t = -2.8191, df = 2e+07, p-value = 0.004816
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.0021371829 -0.0003842037
sample estimates:
   mean of x    mean of y 
0.0001410347 0.0014017281 

As you can see the means differ only on the third significant digit.

Because this difference is so small compared to the standard deviation, the predictive power is close to 0 as you can see with a ROC analysis:

r <- pROC::roc(controls = a, cases = b)
auc(r)
Area under the curve: 0.5003

Regarding ROC curves below the diagonal line, this would indicate that your variable is performing worse than random. Usually this happens when you got the labels swapped.