Is it reasonable to run a random forest model with an independent variables that has almost little or no variations. How they impact the model and whats their role in predicting the dependent?from technical point of view I believe they will not be used in building the tress and will have no role in predicting the DV


There's a big difference between "little" and "no" variation.

  • If a variable has little variation, it can still have predictive importance. This should be obvious: if all values of a feature larger than some constant $c$ corresponds to one class, and all other values to the opposite class, it's a really good predictor. (Likewise, if it's completely independent of the class label, then it's worthless.) This is true no matter what the variance of the feature.

  • A variable with no variation cannot predict the outcome. It won't be selected as a feature, since splitting on the feature can never provide information gain. Including (a large number of) zero-variation features can harm the model, however, because random forest randomly selects feature subsets at each split; a large number of invariant features means that the model will not have any information to split on if the random features are all worthless.

| cite | improve this answer | |
  • 1
    $\begingroup$ It's worth mentioning that "little" variation is relative to the units in which the variable is measured. $\endgroup$ – Matthew Drury Mar 19 '18 at 15:52
  • $\begingroup$ Just that the OP didn't mention their concern was indicator variables, and I've seen the "small variance" belief be an issue with new data scientists. $\endgroup$ – Matthew Drury Mar 19 '18 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.