How to deal with linear regression intercepts with high p-values in dichotomic classifier? I have used a simple multivariable logistic regression (as you would get by default glm() with logit in R) in a problem of dichotomic classifier with approx. 100 predictors, i.e. quite a lot of variables in the regression. My approach was quite naive, just wanted to quickly see some classification results, and they were not that bad, about 85% classification accuracy.
Many of the variables' intercepts have reasonably low p-value (<.05) but there are quite a few of them that have p-value >0.6, sometimes even 0.8. I would intuitively say to myself never mind these variables and get rid of them because you can't be sure that the estimated intercepts are reasonably correct. But when I removed randomly couple of the variables whose intercept p-value was high (e.g. >0.5), the eventual classification accuracy dropped (even on validation dataset).
Should I then keep those variables in the model just because of the classification accuracy? Even though it's quite likely their intercept is just 0 (or simply quite different than the one estimated by the model)? Why is this happening? Maybe I'm overfitting the model? Or maybe I'm just freaking out because there's something peculiar in the way how to properly interpret p-values that I don't know about? Or is it just a coincidence that I happened to come across with my particular training/testing datasets, or does this (i.e. leaving out presumably insignificant variables while causing classification deterioration) happen more often in similar situations?
Thank you very much in advance.
 A: The issues that you report are fairly common with large numbers of predictor variables. Multiple regression provides relations of each of your predictor variables to your classification with the other predictor variables taken into account via a linear model. This leads to all sorts of interesting behavior, as seen in many Cross Validated pages including one on a predictor uncorrelated with outcome becoming significant in multiple regression, on two orthogonal predictors affecting each others' significance in multiple regression, on the suppression effect, and even with the sign of a regression coefficient flipping after other predictors are included.
If your main interest is in predicting classification of future cases you are probably better off erring on the side of keeping too many predictors. That's even true for those that do not appear "significant" on their own, as their exclusion may affect the relations of the "significant" variables to the classification and hurt the overall performance of your model, as you have seen.
If the number of cases relative to the number of predictors leads to worries about overfitting or about perfect separation in your logistic regression, you might consider a technique like ridge regression, combined with standard model validation techniques like cross-validation or bootstrapping, to minimize those problems. The Elements of Statistical Learning, freely available, is a good starting point.
