I am just looking to understand how mtry works in random forests. Please correct me if I am wrong.

When you specify mtry (say 10), it takes 10 random variables from your data set and examines them for one tree. So the next tree would take 10 more random variables, examine them, so on and so forth until it runs through the ntrees that you specify and then returns the average estimates for the best/most important variables?


No, that's not how this works.

Consider a single tree being added to a Random Forest (RF) model.

The standard recursive partitioning algorithm would start with all the data and do an exhaustive search over all variables and possible split points to find the one that best "explained" the entire data - reduced the node impurity the most. The data are split according to the best split point and the process repeated in the left and right leaves in turn, recursively, until some stopping rules are met. The key thing here is that each time the recursive partitioning algorithm looks for a split all the variables are included in the search.

Where RF models differ is that when forming each split in a tree, the algorithm randomly selects mtry variables from the set of predictors available. Hence when forming each split a different random set of variables is selected within which the best split point is chosen.

Hence for large trees, which is what RFs use, it is at least conceivable that all variables might be used at some point when searching for split points whilst growing the tree.

  • $\begingroup$ Gavin, in your first paragraph, do you mean "decision tree" rather than "random forest" in the first sentence? (Apologies if I'm daft.) $\endgroup$ – Sycorax Jun 10 '14 at 17:21
  • $\begingroup$ No I meant consider a single tree in an RF model. I should separate it from the next sentence, but the point was, focus on a single tree; if using RPART, it would use all variables for all splits, but in RF the algorithm uses mtry variables selected at random when forming each split. The multiple trees of an RF is irrelevant to the question about mtry and how it factors into the algorithm. $\endgroup$ – Gavin Simpson Jun 10 '14 at 17:37
  • $\begingroup$ Thank you so much! Your explanation has helped me understand this correctly! $\endgroup$ – CooperBuckeye05 Jun 10 '14 at 17:58
  • $\begingroup$ @Gavin Simpson What do you mean by variables in "an exhaustive search over all variables" and in "all the variables are included in the search" ? features/attributes/something else? $\endgroup$ – Christos K. Feb 8 at 14:41
  • $\begingroup$ Yeah, features, attributes, covariates, transformations or functions thereof. The machine learning folks use the term "features" for this. It's basically whatever you have collected or have created from things you collected (in the case of features, think some form of expansion, conversion of some terms to principal components, etc) that you want to use to predict the response. $\endgroup$ – Gavin Simpson Feb 8 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.