$R^2$ (OOB) worse than $R^2$ (test) I have fitted a regression random forest model using a data set split (80/20) for training and testing. The resulting model gives me an $R^2$ (OOB) of 0.21, and the $R^2$ computed on the testing is 0.82! That is a huge difference and I am unsure whether my model explains 21% or 82% of the variability in the response. Should I believe the OOB $R^2$?
 A: Besides r2 not being mathematically valid for non-linear regressions, it is still widely used for model validation.
R2 is a funny thing and only looking at r2 might lead you to wrong conclusions (even for linear models). The problem with r2 is that you divide the sum of squared residuals by the sum of squared error to the mean.
This means if your target is not so wide spread (e.g.: assume true y is only between 0 and .2) then a straight line with the mean value of y could be better than your predictions using some sophisticated machine learning models (at least in theory).
But this also could mean that small changes in your predictions can have big a big impact on your r2.
I would suggest the following:

*

*Add at least one more evaluation metric: Mean absolute error is good, because you can have a good sense of its meaning. Also try to get root mean squared error (any values below 1 are great)


*Plot y against your predictions (if possible for your OOB sample, too. --> Not so sure this is available in sklearn)


*Check that your data got sampled (I guess you allowed random sampling in train-test, but just to be sure I am mentioning it)


*Try to change some hyperparameters of your random forest and see if you get the same difference --> reduce max_depth for example
I am pretty sure that this looks like some kind of bug and you should be able to solve this. If the data gets split randomly it should not be possible to see such a big difference, although the only reason could be a very small sample size, where some outliers have a big impact on your performance.
