I have a dataset with 163 observations (all countries in the world with population > 1000000) and 290 variables related to their disease burden and performance. Because there are more variables than observations I cannot run a standard linear regression. Therefore I tried both a random forest and a LASSO regression. Both give different variables importances. Which one is the most reliable in this case?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ Possible duplicate of Can Random Forest be used for Feature Selection in Multiple Linear Regression? $\endgroup$– Sycorax ♦Commented Jun 24, 2016 at 14:01
-
$\begingroup$ That thread certainly provides some helpful information, @GeneralAbrial, but I don't think it's a duplicate. $\endgroup$– gung - Reinstate MonicaCommented Jun 24, 2016 at 14:22
-
$\begingroup$ There are high chances you might be overfitting. What is the depth of tree you are using in RF? What is the number of iterations you are using? Remember RF doesn't make a distinction between correlated variables, so check your Variance inflation before feeding into RF. Also it could be pointing to some non-linearity in this data. $\endgroup$– user2542275Commented Jun 24, 2016 at 14:24
-
$\begingroup$ Why would you expect two different algorithms to give the same result? Random forest is not estimating a linear model, while LASSO is. The difference is very important, and the link provides a thoroughly worked example of why. The only novel question here is "which is the most reliable?" but that's unanswerable: reliable for what? Reliable for tree models or reliable for linear models? Reliable in the sense that you minimize predictive risk? Reliable in the sense that model importances are statistically consistent? $\endgroup$– Sycorax ♦Commented Jun 24, 2016 at 14:56
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
Before going deeper into the comparison make sure that each of the two methods agrees with itself. You can find this out by bootstrapping the entire variable importance process a few times. Plot the original variable importance for each variable vs. the importance estimated from a bootstrap sample.
The bootstrap involves taking samples of size $n$ with replacement from the original dataset of $n$ observations, and repeating any analysis. The repetitions have to be "from scratch." Here is what the process looks like in R:
n <- NROW(mydata) # mydata = data table, data frame, or matrix
for(i in 1 : 5) {
s <- sample(1 : n, n, replace=TRUE)
f <- whateveranalysis(mydata[s, ])
# Print what you need and look across the 5 bootstraps to
# see the volatility
}
answered Jun 24, 2016 at 12:08
-
$\begingroup$ could you educate me how to do "bootstrapping the entire variable importance process a few times"? Relative link or papers? Thanks $\endgroup$ Commented Jun 24, 2016 at 12:57
-
3$\begingroup$ @hxd1011, Frank has a book (Regression Modeling Strategies) that covers this sort of thing (& more) in great detail. $\endgroup$ Commented Jun 24, 2016 at 14:52
-
2
$\begingroup$
$\endgroup$
No free lunch. You can't really know it beforehand. Unless you have strong reasons to favor one over the other1 you would have to test both solutions to make a proper choice.
1. Say, you really want a linear model and would feel fine with the implications of choosing that over another possible solution that could be actually better, but then you wouldn't even need to test it in the first place.
