How can a glmnet model with no coefficients have perfect performance?

Question

I sometimes run into situations where glmnet appears to be performing well but actually selects zero features. The AUC is near-perfect but the nzero column shows that all the coefficients are zero. How is this possible?

# Load libraries.
library(glmnet)
library(pROC)

# Simulate data.
set.seed(123)
data <- replicate(3, rnorm(50))
colnames(data) <- paste0("Var", 1:3)
outcome <- gl(2, 25, labels = c("sick", "healthy"))

# Test/train Elastic Net models using LOOCV.
results <- lapply(1:nrow(data), function(i) {
  fit <- cv.glmnet(
    x = data[-i, ],
    y = as.numeric(outcome[-i]),
    family = "binomial"
  )
  pred <- predict(
    fit,
    newx = data[i, , drop = F],
    lambda = "lambda.1se"
  )
  data.frame(
    index = i,
    pred = pred[1],
    actual = outcome[i],
    nzero = fit$nzero[fit$lambda == fit$lambda.1se]
  )
})

# Evaluate performance.
results <- do.call(rbind, results)
roc(results$actual, results$pred) # AUC = 1
plot(results$actual, results$pred)
table(results$nzero) # all coefficients are 0

Answer 1

It looks like in your data there is no relationship between the covariates and the outcome. I imagine that the model is discovering that and shrinks the coefficients to 0. If you fit on all the data you see that the intercept is almost 0, which means the model is assigning near 50% probability of belonging to the healthy class. Evaluating the training AUC shows a 50% AUC (as expected).

So what explains the incredible performance on your LOOCV? I think under the hood, roc is doing some magic to always ensure the ROC is >0.5. As you can see, if you do roc(results$actual, -results$pred) (essentially just flipping the sign of the prediction) you will also get an AUC of 1 even though the labels are reversed which should results in an AUC of 0.

Here is an example of what your model is actually doing:

• I choose one observation to exclude. Let's say it is a healthy patient. That means there are 24 healthy patients in the fold and 25 sick in the fold.

• Because there are more sick patients than healthy, and because by construction the outcome and covariates are unrelated, the model's best prediction is that a new case will also be sick. This it gives a negative prediction to the healthy people when they are left out (negative predictions on the log odds scale correspond to predictions smaller than 50%. Because healthy is our positive outcome, this means there is a better chance the held out sample is sick based on the data in the fold).

• A similar argument can be made for sick people. Hold out a sick person and there are more positive cases than negative, leading to a prediction above 0.

Under the assumption that healthy is the positive case, your model should actually have 0 AUC since it is assigning positive cases a smaller risk than negative cases. This should be addressed by flipping the responses. Flipping the predictions should change the ROC, but it doesn't. Thus, I think roc is doing something in the backend.

Answer 2

I agree with @DemetriPananos’s answer and want to suggest a practical solution to anyone in a similar situation. If you’re using pROC::roc, just specify the direction with direction = “>”. This will separate the “real” good results from the “fake” good results (predicting the opposite every time).

roc(results$actual, results$pred, direction = “>”) # AUC = 0

If you’re doing linear regression, I believe that the trend line should always be positive. That’s why I started using R instead of R-squared.

Thank you @DemetriPananos for explaining this to me, it explains a lot of weird behavior I've observed but never understood.