I'm building a binary classifier with logistic regression and boosting. Just like the case that I described in the title, I am a little bit confuse on how to explain the result of those two models together, and how to answer questions like is this variable a key predictor for predicting?
0
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Logistic Regression is a type of generalized linear model, and hence is analogous to linear regression. Because of this, the ability to handle a non-linear relationship is limited with Logistic Regression.
On the other hand, tree-based models can capture non-linear relationships very well.
I think the reason why you're seeing a variable that is important for a tree-based model, but not significant for Logistic Regression, is because that variable has a non-linear relationship with the target (dependent) variable.
$\begingroup$
$\endgroup$
In addition to the nonlinear/linear modeling issues raised in another answer and in the page linked in a comment, this type of behavior might be seen with strictly linear relations in a logistic model if there are correlations among predictor variables.
The first split in a tree will tend to be based on the predictor that has the best single-variable relation to your classification. If there are other predictors that are correlated to that predictor and also are related to the classification, a logistic multiple regression model will consider all those predictors together. In that case the best individually predictive variable may well diminish to statistical "insignificance" in the multiple regression. I have observed such behavior in practice.
To see if this might be happening in your case, examine how the predictors work individually in logistic regression. Also, note that in such instances the choice of "best" variable can be highly sample-dependent; you might examine the behaviors of your models on bootstrap samples of your data (which would occur directly if you were using bagging rather than boosting).
Note that for a predictive logistic model you should not necessarily remove "statistically insignificant" predictors, as they may nevertheless contribute usefully provided that you are not overfitting.
