0
$\begingroup$

I am working with a dataset of 17 predictors and 1000 observations. I am trying to find the most important variables, for which I am using the permutation-based OOB-MSE.

My problem is that each time I run the random forest, the two most important variables remains invariable but the ranking of other less important variables changes, even keeping the command with the same amount of ntree and mtry. Further, I have also noticed that the mean of squared residuals and the % of variance explained also change, not too much, they keep within the range 41.5-43.3% but they change.

I have read that in Random Forest results can change slightly, so maybe this could be normal. However, I am trying to optimize my model using the tuneRF function:

tuneRF(x=datos[,c(9,15,18,20,27,32,38,40,70,73,95,123,131,132,133,134)], 
       y=datos.fin3$dnbr, ntreeTry = 500, stepFactor=2, improve=0.05, 
       trace=TRUE, plot=TRUE, dobest=FALSE)

and each time a run the function the optimized mtry also changes. Could someone tell me if this is normal?

$\endgroup$
2
$\begingroup$

It is completely normal that, with only 1000 observations, your feature importances slightly change at each time you re-train your model.

Each time you retrain the model, different trees will be created with different selected features in it.

Random Forest (and bagging in general) is based a lot on randomization because its aim is to create quite uncorrelated weak-learner (here the trees). It randomizes mostly by

  • using only part of the training set to train a tree
  • using part of the features when choosing where to train
$\endgroup$
  • $\begingroup$ Do you know if the variable importance measures would become asymptotically stable with increasing numbers of trees? Or only the number of observations? $\endgroup$ – mkt - Reinstate Monica Apr 30 '18 at 13:18
  • $\begingroup$ Yes, it should become asymptotically stable with an increasing number of tree. But what I really recommend you to do is to keep a reasonnable number of trees and run the training multiple times. At the end, plot the boxplots of the importance values for each feature, you'll have a good idea of their relative importance $\endgroup$ – Pop Apr 30 '18 at 14:02
  • $\begingroup$ (+1) Good advice - estimating uncertainty in the permutation importance. $\endgroup$ – mkt - Reinstate Monica Apr 30 '18 at 17:46
  • $\begingroup$ First of all, thank you for answaring. Yes the variable importance measures would become asymptotically stable with increasing the number of trees. $\endgroup$ – pau May 2 '18 at 13:49
  • $\begingroup$ What i have done it's what Pop recomends, that is run the training multiple times. However, i still have a doubt, when i try to determine the optimal mtry parameter with the function TuneRF, i also have the same problem, it also change each time. Would you also recommend to run the model several times and plot the boxplots in order to have an idea of the best mtry value? Thank you $\endgroup$ – pau May 2 '18 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.