I typically find that random forest always overfits to some degree on the training data. That is, the in-fit R2 is typically substantially higher than the out-of-fit, cross-validated R2. In general, the only way I can successfully close the gap between these is to severely hamper out-of accuracy. For example, here's a plot showing both 10-fold cross-validated R2 and in-fit R2 (on the training set) as a function of the hyper-parameter "tree depth".
If my goal was to ensure out-of-fit and in-fit R2 were similar, I'd basically have a tree depth of 1. But this is clearly nonsense. The CV R2 saturates around 0.30 in this case, similar to where in-fit R2 saturates. Thus, it seems that even though I am "overfitting" on the training data, for a depth of >20, I see no loss in predictive performance of the model. Sure, the model is confusing noise for signal, but no matter what, noise is noise, so this doesn't really harm performance or predictive mean-square error of the model.
So -- would you even consider this scenario "overfitting"? By definition, overfitting is when out-of-fit performance is poor due to too many variables or poor hyper-parameter choices. So should I not even consider the relative perform of in- vs out-of-fit R2 when considering overfitting? In this example, I've done a bunch of grid searching and variable pruning, and the model robustly has a cross-validted R2 of around 35%. So the overfitting (if you'd call it that) isn't hurting predictive power one way or the other, and, to the contrary, it seems that the model performs best when in-fit "overfitting" is maximized.
Or is this a red flag that something is wrong elsewhere? And if so, what might the problem be?