3
$\begingroup$

I am using SKLearn and Statsmodel in python to build a RF and Logistic Regression, respectively.

I have a feature that the RF indicates is important (feature importance of 0.202, closely behind #1 and #2 most important features).

However, in running the logistic regression, the coefficient associated with this same feature is 0.0009, nearly 0.

What is going on here? Why would splitting on this variable lead to higher information gain in the Random Forest, but contribute little to the logistic regression model?

Improve this question
$\endgroup$
0

4 Answers 4

Reset to default
6
$\begingroup$

The absolute value of the coefficient is not proportional to the importance of the corresponding feature. There are two ways to assess the significance of a given feature in logistic regression (and more generally for Generalized Linear Models):

  • Look at the p-value of this parameter in the output of the logistic regression
  • or: run two models, one with all the features except the feature of interest (the one you want to assess the performance), and run a second model with all the features, including the feature of interest. You can then perform a Likelihood Ratio Test between the two models to see if this feature is significant in the prediction task

The second approach is more reliable.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ This is untrue. The p-value of a parameter is not proportional to the importance of a corresponding feature, either, and neither is the p-value of the likelihood ratio test. Nor is there any guarantee that the variable importance as measured from the results of RF will match the p-value of the feature. Why? Because the p-value has nothing to do with out-of-sample performance whereas the RF importance score does! $\endgroup$
    – Brash Equilibrium
    Commented Aug 28, 2014 at 18:58
5
$\begingroup$

In random forest algorithms, variable importance is not a measure of effect size. It is a measure of the contribution of a variable to out-of-bag predictive performance. In random forest, a variable can be important due to the way it interacts with other variables, and to the way it separates the data on its own. Nothing prevents a variable with a small linear effect size as estimated in a logistic regression model from having high importance in a random forest fit. Indeed, nothing prevents the inclusion or exclusion of a variable with a small effect size from having a large effect on predictive accuracy even within the framework of logistic regression, especially if there are strong confounding or mediation effects of that variable on other predictors, and including those confounding or mediation effects leads to better predictions.

Improve this answer
$\endgroup$
4
$\begingroup$

Consider the following situation : suppose that your feature $X$ has a coefficient of $t$ after training. Now consider the same model, replacing $X$ by $2X$.

Then your coefficient will now be $t/2$. However the relative importance of the feature hasn't changed at all, since the information contained by $X$ hasn't changed a bit when multiplying it by $2$.

The coefficients in logistic regression just aren't a good measure of the importance of the feature (although they can be an indication if the features are scaled), it is preferable to use other metrics like comparing likelihoods when omitting the features.

Improve this answer
$\endgroup$
0
$\begingroup$

This is similar to a question I asked a while back- Mismatch between significant variables from logistic regression and tree splits in R

Hopefully one of the answers will be useful to you.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.