Preprocessing: Why do we remove constant/invariant predictors?

For a recent research project we used machine-learning. In the preprocessing phase we removed 2 predictors because they contained mostly uninformative 0 values (x1 = 100%, x2 = 99%). Is it the right terminology to call them invariant?

My main justification for the exclusion is, that it reduces the degrees of freedom and therefore the risk of overfitting.

I am looking for a paper / source to justify this procedure, besides the face validity of doing so. Do you have a recommendation for such a source?

Thanks to mkt:
I found what I was looking for in this book: Kuhn, M., & Johnson, K. (2013). Applied predictive modeling. Applied Predictive Modeling (Vol. 26). New York: Springer. https://doi.org/10.1007/978-1-4614-6849-3

• Suppose $x_2$ is an indicator of gender (0=male, 1=female) in an almost predominantly male population (99% zeros) and the response is whether a subject was ever pregnant. Would you still want to drop $x_2$ from the model?? – whuber Jul 16 '19 at 13:54
• Thanks for pointing this out. I think you are right, so of cause it is important to consider the data structure and the purpose of the analysis. In our case it is not a dummy coded variable. It is the probabilty of a feature is being active. So in the case of x2 there was only one unique value of 0.01, meaning a 1% chance that this feature was active. I think it is justified to drop it. – PythonBeginner Jul 16 '19 at 14:19
• That's fine--it was exactly my intention that you conclude judgment is needed. The point is that we cannot expect to find any authority to justify throwing out any explanatory variable based on how close it comes to being constant. Indeed, any source that might make a blanket statement of that nature should be discredited in light of the example I offered. – whuber Jul 16 '19 at 14:33

1 Answer

When you are fitting a regression-type model, you are trying to explain or predict variation in one variable with variation in others. If all values are identical, there is no variation for you to work with! There is no meaningful information in there to use, either as a predictor or a target/dependent variable. So you are completely justified in excluding it.

This is a very basic point, and so I don't think you need a citation to justify it. But if you do, I'd suggest looking at your favourite introductory text - it is probably mentioned there.

This would be different in the case where values are not 100% identical. If you have a few different values in there, then excluding the variable may or may not be justified depending on the question and the nature of the data.

• I would argue that in the case were 99% of the values are identical it is also problematic if you use resampling (i.e. k-Fold CV). Anyways thanks for the tip with the introductory text. I tried to search for it in an Introduction to statistical learning (James et al.). But didn't found it. However I went through my books and found something in Kuhn, M., & Johnson, K. (2013). Applied predictive modeling. Applied Predictive Modeling (Vol. 26). New York: Springer. doi.org/10.1007/978-1-4614-6849-3 – PythonBeginner Jul 16 '19 at 14:22