I am observing an increase in the predictive power of my logistic regression when I remove certain predictors. It might be a bug in my code, so I'm wondering: is this statistically possible? If so, what are the possible explanations?

- there is strong multicollinearity in my predictors
- I am not interested in interpreting my parameter values, I just want high predictive power - I have 15 regressors and a dataset size of 100k (10k for validation to avoid overfitting, 10k for testing)
- I am using a multi-layer perceptron (neural network)
- I measure predictive power by % of test cases correctly classified

  • $\begingroup$ How are you measuring predictive power? If it's an out-of-sample estimate, removing a predictor could increase it. $\endgroup$ – Scortchi - Reinstate Monica Feb 18 '16 at 23:03
  • 1
    $\begingroup$ yes, if some of the predictors had less predictive power. also if you have too less samples, reducing the number of predictors can help in preventing overfitting. $\endgroup$ – jeff Feb 18 '16 at 23:03
  • $\begingroup$ Certainly possible then - that's the point of having a test set. One rule of thumb for observational data suggests over-fitting is likely to be a problem when you've less than ten observations of the minority class for every degree of freedom in your model. Your test set might also not be giving a good estimate of out-of-sample performance if the minority class is small - consider resampling validation -; & classification accuracy isn't the most stable measure - consider a proper scoring rule or area under the ROC curve. $\endgroup$ – Scortchi - Reinstate Monica Feb 19 '16 at 10:42
  • 1
    $\begingroup$ You are using a discontinuous improper accuracy scoring rule. See the Information Loss chapter in Biostatistics for Biomedical Research available from biostat.mc.vanderbilt.edu/ClinStat where you'll see an example where adding a very important predictor makes the % "classified" "correctly" go down. $\endgroup$ – Frank Harrell Feb 19 '16 at 21:43
  • 1
    $\begingroup$ How exactly are you determining predictive power? If you haven't taken steps to assure that the training data are representative of the test data, then there's no guarantee you'll achieve any degree of predictive power. $\endgroup$ – whuber Feb 19 '16 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.