I have fitted a regression random forest model using a data set split (80/20) for training and testing. The resulting model gives me an $R^2$ (OOB) of 0.21, and the $R^2$ computed on the testing is 0.82! That is a huge difference and I am unsure whether my model explains 21% or 82% of the variability in the response. Should I believe the OOB $R^2$?
$\begingroup$
$\endgroup$
2
-
2$\begingroup$ Note that in a nonlinear model like a random forest, $R^2$ lacks its usual “proportion of variance explained” interpretation. $\endgroup$– DaveCommented Dec 6, 2021 at 23:55
-
$\begingroup$ Aside @Dave's reasonable comment ( +1) can you please report MSE and MAE for testing and OOB? $\endgroup$– usεr11852Commented Dec 16, 2021 at 21:40
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
19
Besides r2 not being mathematically valid for non-linear regressions, it is still widely used for model validation.
R2 is a funny thing and only looking at r2 might lead you to wrong conclusions (even for linear models). The problem with r2 is that you divide the sum of squared residuals by the sum of squared error to the mean.
This means if your target is not so wide spread (e.g.: assume true y is only between 0 and .2) then a straight line with the mean value of y could be better than your predictions using some sophisticated machine learning models (at least in theory).
But this also could mean that small changes in your predictions can have big a big impact on your r2.
I would suggest the following:
Add at least one more evaluation metric: Mean absolute error is good, because you can have a good sense of its meaning. Also try to get root mean squared error (any values below 1 are great)
Plot y against your predictions (if possible for your OOB sample, too. --> Not so sure this is available in sklearn)
Check that your data got sampled (I guess you allowed random sampling in train-test, but just to be sure I am mentioning it)
Try to change some hyperparameters of your random forest and see if you get the same difference --> reduce max_depth for example
I am pretty sure that this looks like some kind of bug and you should be able to solve this. If the data gets split randomly it should not be possible to see such a big difference, although the only reason could be a very small sample size, where some outliers have a big impact on your performance.
-
5$\begingroup$ $R^2$ is funky in the nonlinear case, but "not...mathematically valid for non-linear regressions" takes it too far. $R^2$ is just as valid as $MSE$. $\endgroup$– DaveCommented Dec 16, 2021 at 21:06
-
3$\begingroup$ Minimizing (r)mse is equivalent to maximizing the “invalid” $R^2$. $\endgroup$– DaveCommented Dec 18, 2021 at 12:50
-
3$\begingroup$ $R^2$ is a monotonic function of MSE. $$R^2=1-\dfrac{nMSE}{\sum_i(y_i-\bar y)^2}$$ $\endgroup$– DaveCommented Dec 18, 2021 at 22:53
-
2$\begingroup$ The denominator is a property of the data. Whether you model with a linear regression, random forest, neural network, or how many times your dog barks when you tell her the values of the features, the denominator is the same. $\endgroup$– DaveCommented Dec 19, 2021 at 21:02
-
4$\begingroup$ Values of RMSE below 1 are great: this is nonsensical, as a change of units can always achieve this. RMSE is 5 metres? Just change to km, and then RMSE is 0.005 km, so that solves the problem. $\endgroup$– Nick CoxCommented Dec 21, 2021 at 14:12