Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm fitting a logistic regression model to predict probabilities from a set of variables. I'm comparing two such models, say M1 and M2. The only difference is that M2 includes all the variables of M1 plus a few more variables. The idea is to see which variables are useful in predicting my dependent variable.

I expected that AUCs should be non-decreasing with the addition of new variables. If the new variables have predictive power, they should increase the AUC, if they don't, then the AUC should be unaffected. But I find that AUC actually decreases as I add a particular set of new variables. What could be the issue here?

I'm using predict() to get the predicted probabilities. Does it automatically drop all the statistically insignificant variables when calculating the predicted value? Could this be the cause of the drop in AUC?

share|improve this question
1  
The most important missing piece of information for me is, whether you are evaluating the AUC on the training data or you use cross validation or a dedicated test set. –  Erik Jun 27 at 10:55
    
I calculate AUC on test data, not training. –  user46768 Jun 27 at 11:17
1  
In this case I recommend reading up on overfitting. There should be some answers around here. –  Erik Jun 27 at 12:27
    
@Erik Ok, I get it now. Thanks. –  user46768 Jun 29 at 4:40

2 Answers 2

up vote 3 down vote accepted

The effect of uninformative features depends largely on your modeling strategy. For some approaches they are irrelevant while for others they can dramatically decrease overall performance.

Your intuition that using more features should necessarily yield a better model is wrong.

share|improve this answer
    
Can you elaborate on why this happens? I have an econometrics background, and if you add variables, you see R-square, a measure of the fit of the model, always go up. Does predict() use the statistically insignificant variables as well when predicting values? That is the only reason why I can think of why this could happen. –  user46768 Jun 27 at 11:20
2  
user46788: The coefficient of determination ($R^2$) doesn't necessarily increase on the test set when you include more predictors. See @Erik's comment. –  Scortchi Jun 27 at 13:21
1  
The reason varies depending on the modelling strategy. Some examples: (1) for distance-based methods such as kNN, adding irrelevant features may have a large impact on the relative distance between instances, (2) for logistic regression, adding features means adding coefficients that need to be estimated (which can degrade the estimates of coefficients related to good features if you don't have enough training instances). –  Marc Claesen Jun 27 at 13:49

Check if you have not missings values in the new variables. Logistic regression reject the cases with missing data, and only adjust the model for full cases. You must sure that you are comparing the discrimination in the same cohorts.

share|improve this answer
    
No, I'm sure I don't have missing values. Both the models are trained on the same dataset and prediction is done on identical test datasets. –  user46768 Jun 27 at 11:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.