Is it correct to evaluate individual drivers with the AUC value?

Question

I have a discussion with my supervisor about the usage of AUC to determine, basically, the importance of three different drivers consisting of multiple variables each. He claims I can look into the AUC value for the overall model and then try to run a similar model only using one driver at a time, obtain the AUC value for each driver and by then assess the importance of each driver.

Inherent driver: 2 variables
Static driver: 2 variables
Dynamic driver: 7 variables

So my AUC output from a binary ElasticNet model would be the following:

Overall AUC score (all drivers included): 0.89

Then I perform the same ElasticNet model but only with my two variables selected in the inherent driver and the dependent variable. And so on with the next drivers etc. etc. The AUC values are as following:

Inherent driver:                          0.58
Static driver:                            0.67
Dynamic driver:                           0.88

1. Does the result then tell me my dynamic driver are relatively more important, or just better at distinguishing 0 from 1?
2. Is this even a statistically sound method? If not how else can I evaluate it?

EDIT:

       V1  dependent  V2    V3    V4     V5    V6     V7    V8   V9     V10   V11
1      -1.3     0     494.  34.1  2.23   43.0  4.58   46.7  283. 0.442  34.5   0
2      -4.2     0     231.  16.9  1.01   69.4  0      66.4  277. 0.959  11.1   0
3     -11.7     0     646.  132.  20.5   88.0  0.063  34.0  291. 5.63   21     0
4      -9.3     0     44.0  16.4  0.397  39.1  2.37   77.6  279. 7.24   31.8   0
5     -14.2     0     88.2  128.  40.6   83.4  1.09   47.2  284. 8.23   2.92   0
6      19.4     0     382.  49.4  1.15   54.4  0.914  53.6  279. 3.03   16.8   1

df <- df %>% select(V1, dependent, V2, V3, V4, V5, V6, V7, V8, V9, V11, V12)
training.samples <- df$dependent %>% createDataPartition(p = 0.8, list = FALSE)
train <- df[training.samples, ]
test <- df[-training.samples, ]
x.train <- data.frame(train[, names(train) != "dependent"])
x.train <- data.matrix(x.train)
y.train <- train$dependent
x.test <- data.frame(test[, names(test) != "dependent"])
x.test <- data.matrix(x.test)
y.test <- test$dependent
list.of.fits.overall.model <- list()
for (i in 0:10){
fit.name <- paste0("alpha", i/10) 
list.of.fits.overall.model[[fit.name]] <- cv.glmnet(x.train, y.train, type.measure = c("auc"), alpha = i/10, family = "binomial", nfolds = 10, foldid = foldid, parallel = TRUE)
}
predicted <- predict(list.of.fits.overall.model[[fit.name]], s = list.of.fits.overall.model[[fit.name]]$lambda.1se, newx = x.test, type = "response")
#PLOT AUC
pred <- prediction(predicted, y.test)
perf <- performance(pred, "tpr", "fpr")
plot(perf)
abline(a = 0, b = 1, lty = 2, col = "red")
auc_ROCR <- performance(pred, measure = "auc")
auc_ROCR <- [email protected][[1]]
auc_ROCR

Now I repeat the entire elastic-net modeling procedure (search for optimal ridge/lasso tradeoff and optimal penalty value) with just two variables. Bascially, I change the following:

df.inherent <- df %>% select(V1, dependent, V2)
training.samples <- df.inherent$dependent %>% createDataPartition(p = 0.8, list = FALSE)
train <- df.inherent[training.samples, ]
test <- df.inherent[-training.samples, ]
x.train <- data.frame(train[, names(train) != "dependent"])
x.train <- data.matrix(x.train)
y.train <- train$dependent
x.test <- data.frame(test[, names(test) != "dependent"])
x.test <- data.matrix(x.test)
y.test <- test$dependent
list.of.fits.inherent <- list()
for (i in 0:10){
fit.name <- paste0("alpha", i/10) 
list.of.fits.inherent[[fit.name]] <- cv.glmnet(x.train, y.train, type.measure = c("auc"), alpha = i/10, family = "binomial", nfolds = 10, foldid = foldid, parallel = TRUE)
}
predicted <- predict(list.of.fits.inherent[[fit.name]], s = list.of.fits.inherent[[fit.name]]$lambda.1se, newx = x.test, type = "response")

So eventually, the last thing @EDM questioned in the comments.

Answer 1

Given that penalization is important for your modeling, you are on a potentially good track but you need to incorporate information about the potential error in your quality metric of AUC. You can't compare an AUC of 0.58 against one of 0.67 unless you know how how variable those estimates might be.

A simple way to handle this would be to repeat the process with multiple (say several hundred) test/train splits instead of a single one as you currently perform. Single test/train splits can be unreliable with data sets having anything below several thousand cases. (As you would probably need fewer than 200 cases in the minority class to fit an unpenalized model with 11 predictors reliably, I assume that you don't have several thousand cases and thus should be doing more resampling in any event.) Then you use the variability among the (several hundred) test-set AUC values to gauge whether or not any differences among the predictor subsets are statistically reliable.

You might be better off with a similar approach based on bootstrapping instead of the multiple test/train splits. You first use all of the data to fit a full model. That way you get a full model that, unlike your approach, uses all of the available data to build and doesn't depend on the vagaries of a particular test/train split.

You then repeat the entire modeling process (including choice of alpha and lambda via internal cross-validation) on a few hundred bootstrap samples of the data set, and use the entire data set as the test set in each case. Under the bootstrap principle that is analogous to building models on multiple samples from the entire population of interest and then testing them on the population. You thus get a reasonable estimate of the quality of the modeling process: optimism (overfitting) in the coefficient values, and bias and variability in estimates of your quality measure.

In terms of the modeling, even if you choose to use AUC as your final measure, you should be using the deviance instead of AUC as the criterion for the cross-validation choice of alpha and lambda. The AUC (or C-index) isn't very sensitive for distinguishing between models. Also, think carefully whether the lambda.1se is a good choice in this instance. That helps in getting a parsimonious model, but with so few predictors to start with (only 2 in your second example) you might be much better off with the lambda.min value that minimizes cross-validation error (again, best done with deviance even if your final evaluation needs to be done with AUC).