6
$\begingroup$

NOTE: my limited experience is with Random Forest using R.

Are there any special considerations when using Random Forest (in R) that I should be aware of with respect to the impact of correlated variables or variables derived from other variables in the dataset?

For example if I am trying to predict who might leave our company to go work for another company I might include variables such as the ones listed below. Do I need to be cautious with commingling these variables especially since, for example with Age variable, all are based on the same variable: birthdate? Or rollup fields: Age rolls up to "Age Cohorts" and "Age Cohorts" rolls up to "Age Career Cohort"?

From what I understand Random Forest has feature selection but not sure what this says about correlated fields or fields derived from other fields like the above rollup example.

  • BIRTHDATE BASED VARIABLES (all categorical variables except 'Birth year' and Age)

    1. Age
    2. Age Cohorts (i.e. 20-30, 30-40 yrs old, etc)
    3. Age Career Cohort (similar to above but wider bin i.e ("Early (Age <35)", "Mid (Age 35-49", etc)
    4. Birth year (probably not in R since more than 32 categories)
    5. Generation (i.e. Boomers, Generation X, Y, etc)

  • Hire Date BASED VARIABLES

    1. Years of Service
    2. Years of service chorts

Or even, for example age and service are correlated (r~.57).?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

Classification and regression trees do not have the same type of multicollinearity issues that you have in multiple linear regression. Splits are based on best-split criteria from which you have choices with the Gini index being the most commonly used one. In fact, I think it is beneficial to have highly correlated variables available for selection in building the model. This makes it possible to use good surrogate splits when certain variables used in the constructed tree are missing for a particular data point that you want to predict the outcome for in the case of regression or for the classification of a new case where some covariate is missing.

Now Random Forest creates an ensemble of trees and if variables are highly correlated one may appear in one tree while a variable highly correlated with it may be absent in that particular tree but the situation might reverse for another tree. Since you are doing ensemble averaging and bootstrap bagging I think there is even less of an issue with highly correlated variables in Random Forest than there would be just using CART.

Improve this answer
$\endgroup$
2
  • $\begingroup$ That's what I thought but we were not getting an accurate predication with all the variables we had included in the RF model in R. So we took several variables out. Initially we thought it was because of variables like the ones in my original question. $\endgroup$
    – daniellopez46
    Sep 26, 2012 at 16:25
  • $\begingroup$ But now I am thinking about our binary target and variables that for the minority class vary but for the majority class are all the same. EX: we are predicting if an employee will continue employment with our company or terminate employment. We have a variable called reason that gives reason why an employee left. For the minority class (i.e "leavers") this varies (i.e. "Better career opportunity", "pay not enough", etc). For the majority class (i.e. "stayers") its all labeled the same thing: "stayed". What about these type of fields? $\endgroup$
    – daniellopez46
    Sep 26, 2012 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.