Should you tune 'ntree' in the Random Forest algorithm? In the original paper, I was under the impression that the RF couldn't really overfit.
However, in practice I'm seeing that increasing 'ntree' sometimes increases test set error. Is this due to randomness, or is there an optimal level for 'ntree'? Right now, I'm only using cross-validation to choose the optimal 'mtry'.
I am using R and the randomForest package implementation.
 A: Section 15.3.4 of Elements of Statistical Learning (Hastie et al 2009) (PDF is freely available) discusses this. In short, depending on your point of view, random forest can overfit the data, but not because of ntree.
Hastie et al (2009, page 596) states "it is certainly true that increasing $\mathcal{B}$ [the number of trees] does not cause the random forest sequence to overfit". However, they also state that "the average of fully grown trees can result in too rich a model, and incur unnecessary variance" (op cit.).
In short there may be some overfitting due to the use of fully grown trees (the "average" that they talk about in the second statement), which may be showing up as you add more trees to the forest.
Hastie et al (2009) suggest that this "overfitting" does not often cost much in terms of prediction error, and if you don't tune that parameter then tuning is simplified.
If you want to assure yourself, you could try tuning over mtry plus nodesize and/or maxnodes the latter two which controls the depth of trees fitted.
