2
$\begingroup$

I sometimes run into situations where glmnet appears to be performing well but actually selects zero features. The AUC is near-perfect but the nzero column shows that all the coefficients are zero. How is this possible?

# Load libraries.
library(glmnet)
library(pROC)

# Simulate data.
set.seed(123)
data <- replicate(3, rnorm(50))
colnames(data) <- paste0("Var", 1:3)
outcome <- gl(2, 25, labels = c("sick", "healthy"))

# Test/train Elastic Net models using LOOCV.
results <- lapply(1:nrow(data), function(i) {
  fit <- cv.glmnet(
    x = data[-i, ],
    y = as.numeric(outcome[-i]),
    family = "binomial"
  )
  pred <- predict(
    fit,
    newx = data[i, , drop = F],
    lambda = "lambda.1se"
  )
  data.frame(
    index = i,
    pred = pred[1],
    actual = outcome[i],
    nzero = fit$nzero[fit$lambda == fit$lambda.1se]
  )
})

# Evaluate performance.
results <- do.call(rbind, results)
roc(results$actual, results$pred) # AUC = 1
plot(results$actual, results$pred)
table(results$nzero) # all coefficients are 0
$\endgroup$
8
$\begingroup$

It looks like in your data there is no relationship between the covariates and the outcome. I imagine that the model is discovering that and shrinks the coefficients to 0. If you fit on all the data you see that the intercept is almost 0, which means the model is assigning near 50% probability of belonging to the healthy class. Evaluating the training AUC shows a 50% AUC (as expected).

So what explains the incredible performance on your LOOCV? I think under the hood, roc is doing some magic to always ensure the ROC is >0.5. As you can see, if you do roc(results$actual, -results$pred) (essentially just flipping the sign of the prediction) you will also get an AUC of 1 even though the labels are reversed which should results in an AUC of 0.

Here is an example of what your model is actually doing:

  • I choose one observation to exclude. Let's say it is a healthy patient. That means there are 24 healthy patients in the fold and 25 sick in the fold.

  • Because there are more sick patients than healthy, and because by construction the outcome and covariates are unrelated, the model's best prediction is that a new case will also be sick. This it gives a negative prediction to the healthy people when they are left out (negative predictions on the log odds scale correspond to predictions smaller than 50%. Because healthy is our positive outcome, this means there is a better chance the held out sample is sick based on the data in the fold).

  • A similar argument can be made for sick people. Hold out a sick person and there are more positive cases than negative, leading to a prediction above 0.

Under the assumption that healthy is the positive case, your model should actually have 0 AUC since it is assigning positive cases a smaller risk than negative cases. This should be addressed by flipping the responses. Flipping the predictions should change the ROC, but it doesn't. Thus, I think roc is doing something in the backend.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks @Demetri. I don't know it's an issue with the roc function because the plot looks good as well - all of the healthy patients have a score of +0.04 and all the sick patients have a score of -0.04. I do LOOCV this way because I want to save the coefficients on each loop. $\endgroup$ – Daniel Freeman Jul 4 at 18:07
  • $\begingroup$ @DanielFreeman I've added some detail to my answer. If you still disagree that roc is doing something to make your results have AUC>0.5 consider the fact that any model with an AUC<0.5 can be improved simply by flipping the predictions it makes. If there is still doubt, I'm happy to answer any questions you have. $\endgroup$ – Demetri Pananos Jul 4 at 18:26
  • 3
    $\begingroup$ +1 it's actually an interesting case study where K-fold cross validation seems to break down, always giving an estimated AUC = 0 (questionably flipped to 1 here) when the true AUC is 1/2. AFAICT this problem is not special to leave-one-out (i.e., K-fold CV with K=N), it would be the same for any value of K on average. I tried to search for more info on this example, so far the only reference I've found is in these course notes, see the end of section 43 on p. 72. $\endgroup$ – Jake Westfall Jul 4 at 19:12
  • $\begingroup$ @DemetriPananos I think I'm starting to understand. This is basically a class imbalance problem. In the absence of any useful predictors, cv.glment will make an assumption based on the composition of the training date (how many sick/healthy patients are represented). Since the composition changes slightly depending on the type of sample held out, the predictions are always accurate. Am I on the right track here? $\endgroup$ – Daniel Freeman Jul 5 at 0:02
  • 1
    $\begingroup$ @DanielFreeman If you feel this answer has provided clarity, please consider accepting it. If you don't feel you can accept the answer, please let me know how else I can improve it. $\endgroup$ – Demetri Pananos Jul 5 at 1:45
2
$\begingroup$

I agree with @DemetriPananos’s answer and want to suggest a practical solution to anyone in a similar situation. If you’re using pROC::roc, just specify the direction with direction = “>”. This will separate the “real” good results from the “fake” good results (predicting the opposite every time).

roc(results$actual, results$pred, direction = “>”) # AUC = 0

If you’re doing linear regression, I believe that the trend line should always be positive. That’s why I started using R instead of R-squared.

Thank you @DemetriPananos for explaining this to me, it explains a lot of weird behavior I've observed but never understood.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.