2
$\begingroup$

I have a discussion with my supervisor about the usage of AUC to determine, basically, the importance of three different drivers consisting of multiple variables each. He claims I can look into the AUC value for the overall model and then try to run a similar model only using one driver at a time, obtain the AUC value for each driver and by then assess the importance of each driver.

Inherent driver: 2 variables
Static driver: 2 variables
Dynamic driver: 7 variables

So my AUC output from a binary ElasticNet model would be the following:

Overall AUC score (all drivers included): 0.89

Then I perform the same ElasticNet model but only with my two variables selected in the inherent driver and the dependent variable. And so on with the next drivers etc. etc. The AUC values are as following:

Inherent driver:                          0.58
Static driver:                            0.67
Dynamic driver:                           0.88
  1. Does the result then tell me my dynamic driver are relatively more important, or just better at distinguishing 0 from 1?
  2. Is this even a statistically sound method? If not how else can I evaluate it?

EDIT:

       V1  dependent  V2    V3    V4     V5    V6     V7    V8   V9     V10   V11
1      -1.3     0     494.  34.1  2.23   43.0  4.58   46.7  283. 0.442  34.5   0
2      -4.2     0     231.  16.9  1.01   69.4  0      66.4  277. 0.959  11.1   0
3     -11.7     0     646.  132.  20.5   88.0  0.063  34.0  291. 5.63   21     0
4      -9.3     0     44.0  16.4  0.397  39.1  2.37   77.6  279. 7.24   31.8   0
5     -14.2     0     88.2  128.  40.6   83.4  1.09   47.2  284. 8.23   2.92   0
6      19.4     0     382.  49.4  1.15   54.4  0.914  53.6  279. 3.03   16.8   1

df <- df %>% select(V1, dependent, V2, V3, V4, V5, V6, V7, V8, V9, V11, V12)
training.samples <- df$dependent %>% createDataPartition(p = 0.8, list = FALSE)
train <- df[training.samples, ]
test <- df[-training.samples, ]
x.train <- data.frame(train[, names(train) != "dependent"])
x.train <- data.matrix(x.train)
y.train <- train$dependent
x.test <- data.frame(test[, names(test) != "dependent"])
x.test <- data.matrix(x.test)
y.test <- test$dependent
list.of.fits.overall.model <- list()
for (i in 0:10){
fit.name <- paste0("alpha", i/10) 
list.of.fits.overall.model[[fit.name]] <- cv.glmnet(x.train, y.train, type.measure = c("auc"), alpha = i/10, family = "binomial", nfolds = 10, foldid = foldid, parallel = TRUE)
}
predicted <- predict(list.of.fits.overall.model[[fit.name]], s = list.of.fits.overall.model[[fit.name]]$lambda.1se, newx = x.test, type = "response")
#PLOT AUC
pred <- prediction(predicted, y.test)
perf <- performance(pred, "tpr", "fpr")
plot(perf)
abline(a = 0, b = 1, lty = 2, col = "red")
auc_ROCR <- performance(pred, measure = "auc")
auc_ROCR <- auc_ROCR@y.values[[1]]
auc_ROCR

Now I repeat the entire elastic-net modeling procedure (search for optimal ridge/lasso tradeoff and optimal penalty value) with just two variables. Bascially, I change the following:

df.inherent <- df %>% select(V1, dependent, V2)
training.samples <- df.inherent$dependent %>% createDataPartition(p = 0.8, list = FALSE)
train <- df.inherent[training.samples, ]
test <- df.inherent[-training.samples, ]
x.train <- data.frame(train[, names(train) != "dependent"])
x.train <- data.matrix(x.train)
y.train <- train$dependent
x.test <- data.frame(test[, names(test) != "dependent"])
x.test <- data.matrix(x.test)
y.test <- test$dependent
list.of.fits.inherent <- list()
for (i in 0:10){
fit.name <- paste0("alpha", i/10) 
list.of.fits.inherent[[fit.name]] <- cv.glmnet(x.train, y.train, type.measure = c("auc"), alpha = i/10, family = "binomial", nfolds = 10, foldid = foldid, parallel = TRUE)
}
predicted <- predict(list.of.fits.inherent[[fit.name]], s = list.of.fits.inherent[[fit.name]]$lambda.1se, newx = x.test, type = "response")

So eventually, the last thing @EDM questioned in the comments.

$\endgroup$
5
  • $\begingroup$ Is there something important about using AUC rather than another performance metric, say classifier accuracy or Brier score? $\endgroup$
    – Dave
    Aug 17, 2020 at 17:53
  • $\begingroup$ Well, does it matter if I changed it to Brier Score but the outline of the question is the same? Let's say the Brier Score would be 0.008 overall, and it becomes larger for inherent and static but around the same for dynamic. I would not say it change my question substantially. And, I have read about the proper scoring rules before. I decided to go with AUC. $\endgroup$
    – Thomas
    Aug 17, 2020 at 19:07
  • $\begingroup$ Please say more about what you mean when you say things like "perform the same ElasticNet model but only with my two variables selected." Are you using your original elastic net model and only using those 2 variables for prediction, effectively setting all the other variables to 0 (or to the reference level for a categorical predictor)? Or are you repeating the entire elastic-net modeling procedure (search for optimal ridge/lasso tradeoff and optimal penalty value) with just those two variables? Or something else? $\endgroup$
    – EdM
    Aug 18, 2020 at 21:49
  • $\begingroup$ @EdM, sorry I meant the latter in which I repeat the entire elastic-net with just the two variables. I updated my question. Thank you. $\endgroup$
    – Thomas
    Aug 19, 2020 at 9:06
  • $\begingroup$ That extra information in the question helps a lot. $\endgroup$
    – EdM
    Aug 19, 2020 at 16:42

1 Answer 1

3
$\begingroup$

Given that penalization is important for your modeling, you are on a potentially good track but you need to incorporate information about the potential error in your quality metric of AUC. You can't compare an AUC of 0.58 against one of 0.67 unless you know how how variable those estimates might be.

A simple way to handle this would be to repeat the process with multiple (say several hundred) test/train splits instead of a single one as you currently perform. Single test/train splits can be unreliable with data sets having anything below several thousand cases. (As you would probably need fewer than 200 cases in the minority class to fit an unpenalized model with 11 predictors reliably, I assume that you don't have several thousand cases and thus should be doing more resampling in any event.) Then you use the variability among the (several hundred) test-set AUC values to gauge whether or not any differences among the predictor subsets are statistically reliable.

You might be better off with a similar approach based on bootstrapping instead of the multiple test/train splits. You first use all of the data to fit a full model. That way you get a full model that, unlike your approach, uses all of the available data to build and doesn't depend on the vagaries of a particular test/train split.

You then repeat the entire modeling process (including choice of alpha and lambda via internal cross-validation) on a few hundred bootstrap samples of the data set, and use the entire data set as the test set in each case. Under the bootstrap principle that is analogous to building models on multiple samples from the entire population of interest and then testing them on the population. You thus get a reasonable estimate of the quality of the modeling process: optimism (overfitting) in the coefficient values, and bias and variability in estimates of your quality measure.

In terms of the modeling, even if you choose to use AUC as your final measure, you should be using the deviance instead of AUC as the criterion for the cross-validation choice of alpha and lambda. The AUC (or C-index) isn't very sensitive for distinguishing between models. Also, think carefully whether the lambda.1se is a good choice in this instance. That helps in getting a parsimonious model, but with so few predictors to start with (only 2 in your second example) you might be much better off with the lambda.min value that minimizes cross-validation error (again, best done with deviance even if your final evaluation needs to be done with AUC).

$\endgroup$
3
  • $\begingroup$ Hey EdM, first and foremost thank you for your very enlightening answer. I have now processed it all, and I have used the pROC package which has a ci.auc part in which data is bootstrapped, resulting in an AUC with 95 % confidence interval. Would you accept this as a better solution than what I have previously put forward in my question? $\endgroup$
    – Thomas
    Sep 11, 2020 at 9:00
  • $\begingroup$ @Thomas I'm not familiar with the pROC package so I can't say for sure. If the entire modeling process is repeated within each of the bootstrap resamples and AUC is then calculated on the entire data set, that's about as good as you can get for AUC. Do consider my statements in the last paragraph of the answer about the limitations of AUC as a measure and whether the lambda.1se cross-validation criterion is a good choice, too. $\endgroup$
    – EdM
    Sep 11, 2020 at 14:38
  • $\begingroup$ I did choose the lamda.min thank you. It seems to be statistically sound what I am doing thanks to your answer! $\endgroup$
    – Thomas
    Sep 15, 2020 at 8:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.