Timeline for Unstable variable importance ranking
Current License: CC BY-SA 3.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 23, 2015 at 11:41 | comment | added | Alex_cgn | Yes I think something must be wrong, because when I run the model with the 60 input variables, only the first 4 variables stay the same and afterwards the order changes slightly. The input variables are uncorrelated, so there shouldn't be an issue with that. And I even ran the model with ntree=8000 (my best mtry is 20), so that should not be a problem either...But my R² is still really low, so maybe this has an influence...Do you have any other idea? | |
| Apr 23, 2015 at 7:38 | comment | added | Antoine | OK so it seems that you have two covariates that are really contributing to the predictive accuracy, 3 that have close to zero importance, and some mildly important input variables in the middle. Do these results carry over when you use the full set of 60 predictors? What matters the most is the relative importance of the variables. If you observe a lot of variations among variables that are not important at all compared to the top ones, you don't care, it's probably just noise/randomness. If you observe variation for variables that do have some relative importance, something may be wrong. | |
| Apr 23, 2015 at 6:44 | vote | accept | Alex_cgn | ||
| Apr 28, 2015 at 7:18 | |||||
| Apr 23, 2015 at 6:43 | vote | accept | Alex_cgn | ||
| Apr 23, 2015 at 6:44 | |||||
| Apr 23, 2015 at 6:41 | comment | added | Alex_cgn | Great thank you for your help @user2835597. I ran the model with a subset of 10 input variables (and 340 observations) and you are right, the first two most important variables stay the same as well as the three last variables. Only the ones in between change. Then I should probably derive that the top variables are important, but I should not insist on the order of the variables, right? And regarding the measurement, I use both importance and varImpPlot and for both of them I always get the same results, so I guess that is fine... | |
| Apr 22, 2015 at 16:19 | comment | added | Antoine | I just saw that you are using a mix of binary and continuous variables. The binary variables should maybe be entered as factors, to make sure that RF differentiate between them and the continuous ones | |
| Apr 22, 2015 at 16:15 | comment | added | Antoine |
Your commands seem fine to me. How much observations and predictors do you have? How do you get the importance measures? importance? varImpPlot? Especially, for the latter, see the sort, scale, and n.var parameters. If your covariates are really doing a poor job predicting the outcome, or if your data is very noisy, that could explain the variability. But I would expect at least one predictor or two to stay always on top. What is the amount of variation you observe from run to run? Is it totally random?
|
|
| Apr 22, 2015 at 15:57 | comment | added | Alex_cgn | Yes I always use tuneRF to optimize my model and I even tried to run it with, e.g. 2000 trees: bestmtry <- tuneRF(D[,2:NCOL(D)],D$y, ntreeTry=2000, stepFactor=1.5,improve=0.01, trace=TRUE, plot=TRUE, dobest=FALSE) rf<-randomForest(y~.,data=D, ntree=2000, mtry=22, importance=T, proximity=T, replace=T, do.trace=T, keep.forest=T, na.action="na.omit"). Is there maybe an issue in my command? Furthermore, even the optimal model has a low R², so could be the reason for the changing of the ranking that my model just doesn't explain the dependent variable well? | |
| Apr 22, 2015 at 15:37 | history | answered | Antoine | CC BY-SA 3.0 |