2
$\begingroup$

(I am new to machine learning so please bare with me) I am using Random Forest Regression algorithm but I am seeing interesting results. I randomly split data into validation set, test set, and training set, and I used the Python tool GridSearchCV to tune Random Forest, and no matter how I tune, the most complicated RF had the highest score on both validation and test sets no matter how I tune its complexity mathematically (although training and test accuracy differed by 5 to 7 percent). Could this possibly happen if the problem is simple (simple enough that it can train well using fewer points) and the model is training on huge load of data?

$\endgroup$
7
  • $\begingroup$ Could you pls elaborate on "...always output best scores for all of these three sets" ? $\endgroup$ Jul 14, 2018 at 22:53
  • $\begingroup$ the most complicated RF had the highest score on both validation and test sets no matter how I tune its complexity mathematically $\endgroup$
    – Ted
    Jul 14, 2018 at 22:57
  • $\begingroup$ What does "complicated" mean in your context. $\endgroup$ Jul 15, 2018 at 1:25
  • $\begingroup$ Depth of trees @Matthew Drury $\endgroup$
    – Ted
    Jul 15, 2018 at 1:46
  • $\begingroup$ How do you know the problem is simple? Did you construct it yourself? $\endgroup$
    – jbowman
    Jul 15, 2018 at 16:27

1 Answer 1

1
$\begingroup$

This is actually the way Random Forest is designed to work, so what you are seeing is not too surprising if you know a few things about the algorithm.

It helps to think in terms of the bias variance tradeoff. As a reminder, at least roughly:

$$ \text{Test Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Error} $$

A random forest is an average of decision trees. The individual trees are decorated from one-another by clever subsampling schemes: bootstrap sampling of the training data, and split level subsampling of the features.

Since any the trees are identically distributed (each individual possibility for a fit tree is just as likely to show up as the 67th one in a forest as the 154th one) the bias of the average of all the trees is the same as for a single tree. So the averaging keeps the bias of the model the same.

On the other hand, since the trees are not perfectly correlated, the variance of the average of the ensemble is less than that of a single tree.

This is how random forest works, it attacks the variance part of the error decomposition. But for this to work out, you need the bias of a single tree to be low. Having multiple trees will not make it any lower, so your only chance to keep the bias low is in the complexity of the component trees.

Thus, in many cases, the best strategy for building a forest is just to build low bias, i.e. very deep, trees, and let the averaging attack the incurred variance. It is not always the optimal strategy to grow the trees as deep as possible, it's conceivable that averaging does not overcome the suffered variance, but it's not particularly surprising when it does work out that way.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.