I am new to machine learning. I applied logistic regression and random forest on a same dataset. So I get variable importance (absolute coefficient for logistic regression and variable importance for random forest). I am thinking to combine the two to get a final variable importance. Can anyone share his/her experience? I've checked bagging, boosting, ensemble modeling, but they are not what I need. They are more of combining information for the same model across replicates. What I am looking for is to combine result of multiple models.
It probably depends on what you want to use variable importances for. Is it to be used as a criterion for feature selection for a third classification model? In that case you could try to compute a weighted average the variable importances (maybe after normalizing each individual variable importance vector to unit length) for various values and the averaging weight and then pickup the value that yields the best cross-validated score for the final model.
As for combining the outcome of the logistic regression model and the random forest model (without considering variable importances), the following blog post is very informative and demonstrates that a single averaging of the output is a simple yet very effective ensemble method for regression models.
(Commenting on above response and feedback)
Thanks for reading the blog!
The cross-entropy error function has a little cheat, truncating predicted values to [1e-10, 1-1e-10] as a cheap and easy way to prevent errors in the log functions. Otherwise, this is the standard formula.
For the dataset, it is very possible to have datasets where a random forest is far superior to a log. reg. and the log. reg. adds nothing to the ensemble. Make sure, of course, that you are using hold-out data - a random forest will almost always have superior results on the training data due to having far more effective parameters.