# Can AUC decrease with additional variables?

I'm fitting a logistic regression model to predict probabilities from a set of variables. I'm comparing two such models, say M1 and M2. The only difference is that M2 includes all the variables of M1 plus a few more variables. The idea is to see which variables are useful in predicting my dependent variable.

I expected that AUCs should be non-decreasing with the addition of new variables. If the new variables have predictive power, they should increase the AUC, if they don't, then the AUC should be unaffected. But I find that AUC actually decreases as I add a particular set of new variables. What could be the issue here?

I'm using predict() to get the predicted probabilities. Does it automatically drop all the statistically insignificant variables when calculating the predicted value? Could this be the cause of the drop in AUC?

-
The most important missing piece of information for me is, whether you are evaluating the AUC on the training data or you use cross validation or a dedicated test set. – Erik Jun 27 '14 at 10:55
I calculate AUC on test data, not training. – user46768 Jun 27 '14 at 11:17
In this case I recommend reading up on overfitting. There should be some answers around here. – Erik Jun 27 '14 at 12:27
@Erik Ok, I get it now. Thanks. – user46768 Jun 29 '14 at 4:40

The effect of uninformative features depends largely on your modeling strategy. For some approaches they are irrelevant while for others they can dramatically decrease overall performance.

Your intuition that using more features should necessarily yield a better model is wrong.

-
Can you elaborate on why this happens? I have an econometrics background, and if you add variables, you see R-square, a measure of the fit of the model, always go up. Does predict() use the statistically insignificant variables as well when predicting values? That is the only reason why I can think of why this could happen. – user46768 Jun 27 '14 at 11:20
user46788: The coefficient of determination ($R^2$) doesn't necessarily increase on the test set when you include more predictors. See @Erik's comment. – Scortchi Jun 27 '14 at 13:21
The reason varies depending on the modelling strategy. Some examples: (1) for distance-based methods such as kNN, adding irrelevant features may have a large impact on the relative distance between instances, (2) for logistic regression, adding features means adding coefficients that need to be estimated (which can degrade the estimates of coefficients related to good features if you don't have enough training instances). – Marc Claesen Jun 27 '14 at 13:49

Check if you have not missings values in the new variables. Logistic regression reject the cases with missing data, and only adjust the model for full cases. You must sure that you are comparing the discrimination in the same cohorts.

-
No, I'm sure I don't have missing values. Both the models are trained on the same dataset and prediction is done on identical test datasets. – user46768 Jun 27 '14 at 11:21