6
$\begingroup$

I have a large number of variables that I'm trying to reduce, and I've stumbled on Kuhn's (2008) suggestion to eliminate variables with zero or near-zero variance:

[Near-zero variance means that the] fraction of unique values over the sample size is low (say 10%) [...] [and the] ratio of the frequency of the most prevalent value to the frequency of the second most prevalent value is large (say around 20). If both of these criteria are true and the model in question is susceptible to this type of predictor, it may be advantageous to remove the variable from the model.

-- Kuhn, M., & Johnson, K. (2013). Applied predictive modeling, New York, NY: Springer.

This makes sense to me, it's been used in other studies, and it would do exactly what I'm hoping for with my data. However, as far as I can tell, Kuhn doesn't provide any justification (either theoretical or empirical) for using this technique, and I can't find any other literature that supports this.

Does anyone know of other sources that demonstrate why this technique works?

$\endgroup$
  • 11
    $\begingroup$ What quantitative criterion do you propose to distinguish "near-zero" from "zero" variance? Although understanding the motivation behind Kuhn's suggestion, I confess to being extremely sceptical, in light of the fact that what matters is the relationship between each explanatory variable and the response variable (after controlling for the other explanatory variables). If tiny changes in one variable consistently are associated with appreciable changes in the response, then surely you would want to keep that variable! $\endgroup$ – whuber Apr 9 '15 at 21:26
  • $\begingroup$ Zero-variance won't even run in most packages/would crash the model. Removing near-zero-variance predictors would also eliminate dummy variables and other binary predictors. this has 2 refs to alternative procedures r-bloggers.com/… $\endgroup$ – katya Apr 9 '15 at 21:30
  • 1
    $\begingroup$ There may be that danger, yes--but eliminating the variable altogether is the wrong way to deal with it (in your hypothetical case) because you would then be removing perhaps the most important explanatory variable you have! $\endgroup$ – whuber Apr 16 '15 at 14:50
  • 1
    $\begingroup$ Could you provide a full reference & perhaps a quote or summary of the recommendation? There's probably some sense in it when building a predictive model, but it has to be understood in a context-specific way. A range of temperatures in a sample from 25ºC to 40ºC might be considered a high variance in enzyme kinetics but a low variance in metallurgy. It comes down to not filling your model up with predictors that you'd expect to have a negligible relationship to the response over the range they take in your sample. $\endgroup$ – Scortchi Aug 21 '15 at 10:05
  • 1
    $\begingroup$ @Scortchi Sorry for the delay; obviously haven't been here for a while. :) Here's the info: Near-zero equals: "The fraction of unique values over the sample size is low (say 10%)" and "[t]he ratio of the frequency of the most prevalent value to the frequency of the second most prevalent value is large (say around 20). If both of these criteria are true and the model in question is susceptible to this type of predictor, it may be advantageous to remove the variable from the model." Kuhn, M., & Johnson, K. (2013). Applied predictive modeling New York, NY: Springer. $\endgroup$ – Spencer Greenhalgh Dec 10 '15 at 2:29
3
$\begingroup$

Based on my experience, I often remove near zero variance predictors (or predictors which have one value only) since they are considered to have less predictive power. In some cases such predictors can also cause numerical problems and cause a model to crash. This can occur either due to division by zero (if a standardization is performed in the data) or due to numerical precision issues. The paper (http://www.jstatsoft.org/v28/i05/paper) provides some reasoning though not rigorously proving it in pages 3 and 4.

An example dealing with near zero predictors that I found useful is : https://tgmstat.wordpress.com/2014/03/06/near-zero-variance-predictors/

Hope this helps.

$\endgroup$
  • 6
    $\begingroup$ Predictors with a unique value or with zero variance are constant and therefore will automatically be eliminated by any good regression software. "Near zero," on the other hand, is utterly meaningless, because internally the variables will be standardized and thus all variances become unity (and are all equal). $\endgroup$ – whuber Apr 16 '15 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.