1
$\begingroup$

I have a random forest being applied to 7 different input variables to predict a particular classification. I've done a grid search on the hyperparameters mtry and ntree and it seems as though the algorithm is most accurate when mtry is at 6 (the highest value for mtry I allowed as a hypothetical value in my search). This finding has also been confirmed when applied to a test set. I have an intuition that mtry should always be less than the number of total variables in the model but I can't find anything that explicitly states this.

My question Is there an upper limit to mtry as I think there should be? And if indeed that's the case, what would it indicate if my model gets more accurate as I approach that upper limit? Is that something to be concerned about?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
8
$\begingroup$

mtry is indeed bound by the number of variables in your model, as it specifies the size of the variable subset that is randomly picked for each random forest iteration.

Given that it is a hyper-parameter, there is no way to know ahead of time what would be the best value of mtry. However, values of mtry that are close to the total number of variables in the model may weaken the forest by making the individual decision trees more correlated; when the decision trees consider similar sets of variables to split on, they are more likely to be similar, even if each is fit to a different bootstrapped data set. Ensemble models usually strive for independence of their members, as that improves predictive ability.

Despite these concerns, the value picked by through proper validation, even if it is large, is probably a fair value to consider for a model.

Improve this answer
$\endgroup$
3
  • $\begingroup$ See, I was thinking the same thing but when conducting a grid search for a particular set, I found that the best value of mtry was 2-3x the number of variables. This was replicated when using caret in r to tune mtry. The efficacy of this hyper-parameter value was confirmed through the use of a test dataset. $\endgroup$
    – neanderslob
    Sep 15, 2016 at 0:20
  • 4
    $\begingroup$ Is this using the randomForest package in R? It shouldn't be possible to set mtry > p, due to this block of code in the function: p <- ncol(x) ... if (mtry < 1 || mtry > p) warning("invalid mtry: reset to within valid range") mtry <- max(1, min(p, round(mtry))) Are you seeing this warning when you have mtry > nVariables? $\endgroup$
    – Mir Henglin
    Sep 15, 2016 at 2:41
  • 1
    $\begingroup$ Agree with @MirHenglin. It doesn't make sense to take mtry > p [total number of predictors]. As by definition; 'mtry' is the number of variables randomly sampled as candidates at each split $\endgroup$
    – Dr Nisha Arora
    Dec 13, 2019 at 10:18
2
$\begingroup$

Could the reason for mtry being able to exceed p be because some of the variables are categorical, and automatically one-hot-encoded into several dummy variables?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.