I typically find that random forest always overfits to some degree on the training data. That is, the in-fit R2 is typically substantially higher than the out-of-fit, cross-validated R2. In general, the only way I can successfully close the gap between these is to severely hamper out-of accuracy. For example, here's a plot showing both 10-fold cross-validated R2 and in-fit R2 (on the training set) as a function of the hyper-parameter "tree depth". enter image description here

If my goal was to ensure out-of-fit and in-fit R2 were similar, I'd basically have a tree depth of 1. But this is clearly nonsense. The CV R2 saturates around 0.30 in this case, similar to where in-fit R2 saturates. Thus, it seems that even though I am "overfitting" on the training data, for a depth of >20, I see no loss in predictive performance of the model. Sure, the model is confusing noise for signal, but no matter what, noise is noise, so this doesn't really harm performance or predictive mean-square error of the model.

So -- would you even consider this scenario "overfitting"? By definition, overfitting is when out-of-fit performance is poor due to too many variables or poor hyper-parameter choices. So should I not even consider the relative perform of in- vs out-of-fit R2 when considering overfitting? In this example, I've done a bunch of grid searching and variable pruning, and the model robustly has a cross-validted R2 of around 35%. So the overfitting (if you'd call it that) isn't hurting predictive power one way or the other, and, to the contrary, it seems that the model performs best when in-fit "overfitting" is maximized.

Or is this a red flag that something is wrong elsewhere? And if so, what might the problem be?

  • $\begingroup$ You show results for in-fit and cross validation. Do you also have a hold out dataset to test against? Sometimes data is broken into three groups: training, validation and test ( or other names). The idea is that you might do something like cross validation or other approaches with training and validation but have a final hold out group that has no connection to parameter selection as a final check. How does it do then? $\endgroup$ – Deathkill14 Aug 13 at 10:11
  • $\begingroup$ The validation r2 shows a similar trend to the cross validation r2, with a more gradual slope and peaking near Depth = 30, where it maxes out around r2 = 0.32, about 3-5% lower than the CV r2. $\endgroup$ – user3037237 Aug 13 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.