2
$\begingroup$

I am trying to build a model predicting a binary outcome, say screened vs. non-screened.
A little bit about the data

  1. I have about 40K records. 86% of them have the outcome as screened. It's a very unbalanced data.
  2. And I have about 18 predictors. Most of them have a weak correlation with the outcome. The goal here is to find 3-5 predictors that are the most powerful. I tried two methods
  3. Regular logistic regression. As you may expect, most cases were significant due to the large sample sizes.
>     Coefficients: (1 not defined because of singularities)
>                                       Estimate Std. Error z value Pr(>|z|)    
>     (Intercept)                      1.556e+00  2.703e-01   5.758 8.52e-09 ***
>     medinc                           1.853e-06  9.007e-07   2.057  0.03969 *  
>     medage                          -2.309e-02  2.337e-03  -9.880  < 2e-16 ***
>     raceeth_black                    1.060e+00  4.329e-01   2.449  0.01431 *  
>     raceeth_latino                   5.238e-01  3.767e-01   1.390  0.16444    
>     owner_occ                       -1.613e-02  3.068e-01  -0.053  0.95808    
>     renter_occ                              NA         NA      NA       NA    
>     publicassist                     5.416e-01  2.160e-01   2.508  0.01214 *  
>     nocitizen                       -1.108e+00  2.785e-01  -3.980 6.90e-05 ***
>     no_health_ins                    1.431e-01  2.495e-01   0.573  0.56639    
>     unemployed                      -2.326e-01  5.456e-01  -0.426  0.66983    
>     Mail_Return_Rate_CEN_2010        1.654e-02  3.019e-03   5.479 4.27e-08 ***
>     pct_URBANIZED_AREA_POP_CEN_2010  2.677e-03  5.481e-04   4.884 1.04e-06 ***
>     pct_RURAL_POP_CEN_2010          -4.142e-03  6.974e-04  -5.939 2.87e-09 ***
>     pct_Hispanic_CEN_2010           -6.583e-03  4.178e-03  -1.576  0.11510    
>     pct_NH_White_alone_CEN_2010     -2.833e-03  1.980e-03  -1.431  0.15242    
>     pct_NH_Blk_alone_CEN_2010       -8.922e-03  4.774e-03  -1.869  0.06165 .  
>     pct_Owner_Occp_HU_CEN_2010       8.470e-03  3.156e-03   2.683  0.00729 ** 
>     samp_16                         -8.917e-01  3.428e-02 -26.016  < 2e-16 ***
  1. Then I tried the random forest and printed out the MeanGini by their importance.
>                           V1      MeanDecreaseGini
>     1                   samp_16         825.8772
>     2                    medage         469.3604
>     3 pct_NH_Blk_alone_CEN_2010         466.0336
>     4             no_health_ins         459.0699
>     5                unemployed         452.0088

Both models indicate that variables samp_16 and medage are important or significant, which looks right to me. Logistic regression has more significant variables. However, some significant variables are not shown on the top variables with importance in the random forest model. Two variables like no_health_ins and unemployed, which are the top 5 most important variables in the random forest model, do not even show as significant in the logistic regression.

How should I interpret it? Is it because most variables are too weak? Should I pick the variables that were both significant in the logistic and show importance in the random forest?

And is there anything special we need to deal with the logistic regression when the data is unbalanced? Should I do the undersampling/oversampling first? For the random forest, I try to apply the classwt, but both lead to similar results regarding the importance vector.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ First, if you're concerned with class imbalance, you should address that issue first, using for example SMOTE or similar aglorithms. Then you fit your logistic regression and RF and look for variable importance information. $\endgroup$
    – horaceT
    Commented Sep 14, 2016 at 21:05
  • 2
    $\begingroup$ GLM and RadnomForest are estimating two different things. For an example of why this is important, please see my answer here, which might be a duplicate. stats.stackexchange.com/questions/164048/… $\endgroup$
    – Sycorax ♦
    Commented Sep 14, 2016 at 22:09
  • $\begingroup$ Thanks for pointing me to that post. Very informative. So should I say that variables which show as important in RF but not significant in the logistic regression are possible those with non-linear relation with the outcome variables? $\endgroup$
    – Wei Zeng
    Commented Sep 14, 2016 at 22:43
  • $\begingroup$ Logistic regression have no problems with unbalanced classes, see stats.stackexchange.com/questions/6067/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Sep 7, 2017 at 11:12
  • $\begingroup$ Your classes are that unbalanced either. $\endgroup$
    – David Ernst
    Commented Sep 7, 2017 at 13:28

1 Answer 1

Reset to default
2
$\begingroup$

In my personal opinion....

Thoughts:

  • Random forest looks at nonlinear interactions of variables, but logistic regression does not, unless you explicitly code it.
  • Random forest does bootstrap resampling so it is more resistant to outliers than least-squares fits. You could have ridge or lasso for you logistic regression, but I did not see that indicated.
  • There is a fun library called "Boruta" that has some of the niggling parts of this reasonably engaged.
  • It is sometimes possible to make linear regression as powerful as a random forest by computing the RF variable importances, then associating those with the linear fit as variable weights.

Opinion:

  • I prefer to use the "Z-scores of mean decrease accuracy measure" for variable importance instead of Gini importance.
  • There are cases where the Random Forest has been much more useful for this than logistic regression.
Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Because tree-based methods force a particular type of interaction through the tree structure, they can be less powerful than linear/logistic regression methods when those interactions don't exist or are weak. $\endgroup$
    – jbowman
    Commented Mar 1 at 15:59
  • $\begingroup$ @jbowman - I do not know if that is true. The trivial random forest for classification is a one-tree stump, aka a single logistic regression. If the model was infinitely perfectly classic logistic, then that model is a subset of the random forest model candidates. I can hear that RF might give low but nonzero value for variables that are unimportant as a consequence of bootstrapping... might need to make a question and explore that. $\endgroup$
    – EngrStudent
    Commented Mar 6 at 14:34
  • $\begingroup$ In practice, however, the one-tree stump does not represent the individual trees of "optimal" random forests. And a one-tree stump is not the same as a single logistic regression—consider estimating the probability of contracting cancer as a function of age, sex, body mass index, exercise, and smoking. Even if you only estimate it as a function of age, a spline on age followed by a standard logistic regression will serve you much better than a random forest. $\endgroup$
    – jbowman
    Commented Mar 6 at 15:15
  • $\begingroup$ @jbowman - I am just going to have to run it and see. :) Thanks for the challenge. $\endgroup$
    – EngrStudent
    Commented Mar 7 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.