0
$\begingroup$

I am trying to run a regression estimation on random forest algorithm. As I am very inexperienced in using RF algorithms, I couldn't figure out some questions unless I make a quick survey on literature.

Unless I do know that it's common to take n(parameters)/3 for mtry, I also tried to tuneRF() function of RandomForest package in R. Unfortunately function returned value of 22, while the number of predictors I am using is 38. Although the model I estimated with these parameters gave me lower MAE, I am not sure if a mtry value as high as 22 would cause overfitting.

In addition to this problem I am not sure that growing 500 trees for estimation with 4380 observation, do you have some suggestions?

$\endgroup$
1
$\begingroup$

The default values for mtry (the number of features considered at each split) are there as guidance and a starting point, but they will not necessarily provide the best performance on unseen data.

As you say in the R randomForest package the mtry default for regression is p/3, but if we look at the scikit-learn implementation of RandomForestRegressor we see that the default is p, with other common choices given as sqrt(p) or log2(p), so these defaults are not even necessarily consistent across different implementations of the same algorithm.

The tuneRF() function you have tried attempts to find this best mtry by starting at a given mtry and computes the out-of-bag estimate of error and then iterates to the next mtry, either going up or down by a given step as set in the default or by the user. If the next mtry doesn't improve the error by a specified amount the search ends. The fact that this function has chosen 22 over the default should not concern you, this has been chosen on the basis that this mtry reduces the error on the out of bag samples, which serve as a good approximation of test error. Although if you have started this process at the default you may have missed out more optimal values at the lower range.

Regarding the number of trees, this is also a parameter that can be altered. To attempt to find the optimal mtry and number of trees for your given problem you should really try tuning the model with different parameter combinations over the whole range, testing via cross validation to determine the parameters for best performance. As you have already said you are using R see this walkthrough of this process.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.