Leave insignificant features into logistic regression I am making a model with logistic regression to predict binary sport events and have found some variables that may have an impact. Some features would however not contribute to the accuracy score of the model, but it would also not decrease it. My question is that should I leave these features or just throw them out?
 A: Particularly with logistic regression, it's good to include any features that might reasonably contribute to outcome even if they aren't correlated with other features. As this page shows, there can be a significant bias from omitting potentially informative variables from a logistic regression in situations where omitting them might not bias a standard linear regression.
Remember that lack of statistical signficance doesn't necessarily mean that a variable is unrelated to outcome; it's possible that there is a real relation but you just didn't have enough power in your data and analysis to demonstrate it yet.
In general for predictive models it's best not to exclude features, just be careful that you're not overfitting the model; use a penalized method like ridge regression or LASSO if you are in danger of overfitting. The rule of thumb for logistic regression is that if you have more than 15-20 of the lower-prevalence class per feature in the model you probably won't have problems with overfitting.
