3
$\begingroup$

I am trying to create a logistic regression model and a random forest model on the same data to predict probability of default. For the logistic regression model, I have created some dummy variables from categorical variables. Finally, for the input of logistic regression, I have 9 dummy variables and 2 numeric variables (age and level, age takes values from 18 to 60, level from 4 to 10). I want to use same input dataset for the random forest model. When I did so, using "randomForest" Package, I get following Variable Importance Plot.

enter image description here

Level seems to be a very good variable both by MSE and Node Purity. Also, level is a very important variable in logistic regression (p value ~ 10^-5). However, Age is very important by Node purity, but not by MSE. Also, in logistic regression, age is not a very good variable with p value of 0.026. So I want to understand, Does being numeric increases the node purity importance of a variable by overfitting? Is it not suitable to use numeric and dummy variables together in random forest model? Or is there something I am missing.

I had similar doubts about using numeric and dummy variables in logistic regression, but in logistic regression it did not create any problem.

Improve this question
$\endgroup$
1
  • $\begingroup$ Hi, did you normalise the numerical variables? $\endgroup$
    – Filippo Mazza
    Jun 19, 2017 at 8:39

2 Answers 2

Reset to default
4
$\begingroup$

RF is one of the most robust techniques for handling a combination of data types, yet it can mishandle data in cases when there are data with very few categories (particuarly if they are unbalanced) and many categories. Several options to explore: what is the total amount of variation explained (if very small, the discrepancy is not surprising; and will also tell whether it is indeed 'very good'); are any of the categories unbalanced? are the numerical predictors strongly correlated? have RF setting been optimized? PS What do you mean by "in logistic regression, age is not a very good variable with p value of 0.026"?

Improve this answer
$\endgroup$
4
  • $\begingroup$ Details of RF model: Type of RF: regression, No. of trees: 1000, No. of variables tried at each split: 3, Mean of squared residuals: 0.073,% Var explained: 10.68. I don't understand what you mean by unbalanced categories. Yes there are dummy variables which are 1 for less than 10% of the population. Numerical predictors are not strongly correlated (correlation = -0.03). I don't know how to optimize RF settings. In Logistic regression, every model variable is given an estimate, std error, z value & p value. My understanding is that higher p value of a variable implies lower significance $\endgroup$
    – Gaurav Singhal
    Jul 22, 2015 at 7:37
  • 1
    $\begingroup$ You can disregard IncNodePurity. It is never more useful than %MSE. So the feature age probably has a lot of levels compared to other features. IncNodePurity has a tendency to overestimate importance of feature with a lot of levels. $\endgroup$
    – Soren Havelund Welling
    Jul 23, 2015 at 13:23
  • 1
    $\begingroup$ I see your var.explained is lower than 50%. Then it you can optimize your RF model performance by lowering sampsize to something like 20-60% of your training data size. If N samples is 1000, try sampsize=400. It introduce some more bias and improve variance by 1 to 5%. $\endgroup$
    – Soren Havelund Welling
    Jul 23, 2015 at 13:26
  • $\begingroup$ thanks Soren, your comments are very informative and actually answers my question, I will try the your suggestions, will tell you the results. $\endgroup$
    – Gaurav Singhal
    Jul 25, 2015 at 7:15
4
$\begingroup$

(1) Variable importance is a concept without a set meaning. Very popular in the ML community, but not well developed concept. Your energies would be better focus on concepts such as "all relevant" variables (Boruta) or minimal variable set (VSURF).
(2) If you change the sampling ratio, the importance measures usually change. Worth that try if you have the time :). Standard reference for balanced sampling.
(3) Even ignoring above, the are a fair number of accepted issues with the importance measures. You might want to look into the party package that implements AUC-based importance measures as well as conditional importance measures.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.