I am new to machine learning. I applied logistic regression and random forest on a same dataset. So I get variable importance (absolute coefficient for logistic regression and variable importance for random forest). I am thinking to combine the two to get a final variable importance. Can anyone share his/her experience? I've checked bagging, boosting, ensemble modeling, but they are not what I need. They are more of combining information for the same model across replicates. What I am looking for is to combine result of multiple models.
$\begingroup$
$\endgroup$
2
asked Jan 3, 2013 at 22:05
-
5$\begingroup$ Ensemble modelling can also combine models. Look into majority voting for example. See also, stacking. $\endgroup$– patJan 3, 2013 at 22:14
-
4$\begingroup$ Actually, using the size of coefficients is not a good way to determine "variable importance" in logistic regression. Even if you look at standardized coefficients, that's not a good method. Why? Remember that the coefficients are just estimates and there's an error associated with them. Picking coefficients by size means that you pick those for which you over-estimated the coefficient size and drop those for which you under-estimated the coefficient size. $\endgroup$– user765195Jan 4, 2013 at 4:37
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
It probably depends on what you want to use variable importances for. Is it to be used as a criterion for feature selection for a third classification model? In that case you could try to compute a weighted average the variable importances (maybe after normalizing each individual variable importance vector to unit length) for various values and the averaging weight and then pickup the value that yields the best cross-validated score for the final model.
As for combining the outcome of the logistic regression model and the random forest model (without considering variable importances), the following blog post is very informative and demonstrates that a single averaging of the output is a simple yet very effective ensemble method for regression models.
-
1$\begingroup$ Thank you for your reply. The blog you mentioned is really interesting study. I think I got the idea. Only concern is his cross entropy formula. It seems different with the one I found online. His using: cross.entropy <- function(target, predicted) { predicted = pmax(1e-10, pmin(1-1e-10, predicted)) - sum(target * log(predicted) + (1 - target) * log(1 - predicted)) } $\endgroup$ Jan 4, 2013 at 20:01
-
2$\begingroup$ and when I applied the same idea to my own dataset, I used misclassification error as criteria, the plot is nothing similar. Random forest turns out much better than logistic regression. misclassification error of RF is 0.2, for LR is 0.4. At the same time, AUC for RF is 0.8, for LR is 0.73. $\endgroup$ Jan 4, 2013 at 20:04
$\begingroup$
$\endgroup$
0
(Commenting on above response and feedback)
Thanks for reading the blog!
The cross-entropy error function has a little cheat, truncating predicted values to [1e-10, 1-1e-10] as a cheap and easy way to prevent errors in the log functions. Otherwise, this is the standard formula.
For the dataset, it is very possible to have datasets where a random forest is far superior to a log. reg. and the log. reg. adds nothing to the ensemble. Make sure, of course, that you are using hold-out data - a random forest will almost always have superior results on the training data due to having far more effective parameters.
