0
$\begingroup$

In R, I ran multiple configurations. The weird thing is that increasing nodesize improves the accuracy quite a bit. This is the opposite of what I expected.

nodesize determines the minimal size of the final nodes (leafs). So a bigger number would mean a less deep tree used. Correct?

set.seed(23)
Model4 <- train(Myformula4, method = "rf", 
                data = Train, importance = T, nodesize=1, ntree=100,
                trControl = trainControl(method = "cv", number = 10)))

So this is searching for an optimal mtry (number of features used). Weird thing is I played around with nodesize (1 is advised for classification). Changed it to 50, 90, 100, 150 even 500. And the higher, the more accurate. Any ideas? Details: My data is 25 features, some of which correlated. 25000 rows of data.

$\endgroup$

1 Answer 1

1
$\begingroup$

Did you compare the effect of increasing nodesize on your training error vs. validation error?

It's possible that the "smaller" tree generalizes better. With a nodesize of 1 each terminal node could correspond to a single example, which can perform really well on the training set, but may be overfitting. By increasing the nodesize your trees can't "memorize" the training data.

$\endgroup$
4
  • 1
    $\begingroup$ This is a well known phenomena. In gradient boosting, frequently stumps (tress with just two nodes) perform best. I don't know of a theoretical explanation, but likely the answer has to do with overfitting. $\endgroup$
    – meh
    May 18, 2017 at 18:52
  • $\begingroup$ No, I was only looking at OOB error for the whole dataset. Nothing with test/validation set yet. So I still don't get how a "smaller" tree would be more accurate.... (BTW i'm doing a classifcation with 4 classes). $\endgroup$
    – Cindy88
    May 18, 2017 at 19:04
  • $\begingroup$ Again, with a nodesize of 1, your tree could have each training example in its own terminal node, which is a very large/deep tree. In that extreme case the tree can memorize the correct classification for each training example, but that will generalize terribly to the OOB examples which it hasn't seen yet. By forcing the tree to be smaller (shallower) you're more likely to learn general patterns, vs specific examples. $\endgroup$
    – Max S.
    May 18, 2017 at 19:29
  • $\begingroup$ I see, thank you! Would you say this is more prone to happen when data is noisy? Do the amount of features or their correlation have something to do with the odds of this happening? $\endgroup$
    – Cindy88
    May 18, 2017 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.