Skip to main content

Timeline for $R^2$ (OOB) worse than $R^2$ (test)

Current License: CC BY-SA 4.0

21 events
when toggle format what by license comment
Dec 21, 2021 at 14:17 comment added Nick Cox This mixes much good advice with (in my view) some exaggeration and at least one outright error (see above). My edits are restricted to points of English expression. "not being mathematically valid" is a poor criticism, as R-squared can be well defined, as the answer does. "not a good idea or good method" is a stance that can be explained.
Dec 21, 2021 at 14:14 history edited Nick Cox CC BY-SA 4.0
added 5 characters in body
Dec 21, 2021 at 14:12 comment added Nick Cox Values of RMSE below 1 are great: this is nonsensical, as a change of units can always achieve this. RMSE is 5 metres? Just change to km, and then RMSE is 0.005 km, so that solves the problem.
Dec 21, 2021 at 12:38 comment added janrth @David: I also posted now the question whether mse is valid for non-linear regression while r2 might not be. I hope that will resolve the problem between us:) stats.stackexchange.com/questions/557863/…
Dec 21, 2021 at 12:37 comment added janrth @Dave: r-bloggers.com/2021/03/…
Dec 21, 2021 at 12:30 comment added janrth @Dave: Please read the link I share in this comment. Pseudo-r2 might be used, but it should not be used for model selection. But MSE can be as shown by Ratkowsky in 1990. I don't understand what your point really is. If you want to select your regression model based on r2, then please do so. I also said that people still do it a lot. But my simple example also showed how very wrong an (pseudo-)r2 can be compared to an mse for non-linear functions. Also I pointed to the (probably) most cited paper when it comes to r2 in non-linear systems with thousands of simulations.
Dec 21, 2021 at 10:58 comment added Dave You have the equation for $R^2$ in these comments. If you remain confused about how that is proof of how MSE and $R^2$ are equivalent evaluation metrics, please post a new question. I don’t know where that is addressed explicitly on here, so I would not expect such a question to be closed as a duplicate.
Dec 21, 2021 at 10:33 comment added janrth @Dave: It was not my question in the first place. I just tried to help and said that r-squared is not (mathematically) valid for non-linear functions. This is common knowledge in research I think. Have you read the paper I mentioned above? Also my simple example shows you how unreliable r-squared is for non-linear functions. The question whether r2 is a valid metric for non-linear functions should have been answered here in some other posts, so I don't want to duplicate. But maybe you could point me to research that shows you can use r2 for non-linear functions, so that I can also consider it
Dec 20, 2021 at 22:42 comment added Dave Please post that as a new question and write @Dave when you post back here with a link. You’ve made some (common) mistakes and deserve to have them addressed in a full answer, not just a comment.
Dec 20, 2021 at 22:14 comment added janrth Hey Dave, I created a simple notebook. I hope I did not made any mistakes. Please review, if you feel something is fishy. What you will see is a linear and a non-linear case. In both cases I create the predictions from y+gaussian(mean, std.). And while the mse is very similar in both cases, the r2 is way higher for the non-linear case. I think this is what is often found in literature, that the r2 for non-linear cases is unrealistic high:. Anyway, maybe have a look and tell me what you think: github.com/janrth/r2_non_linearity/blob/master/…
Dec 19, 2021 at 21:02 comment added Dave The denominator is a property of the data. Whether you model with a linear regression, random forest, neural network, or how many times your dog barks when you tell her the values of the features, the denominator is the same.
Dec 19, 2021 at 10:09 comment added janrth And then I think it should be obvious how the relation between the r2 and the mse for the two cases will be different. Maybe I want to try this also now:) I want to see what happens. And again I am not a Mathematician. My understanding of that problem is merely intuitively at the moment. By the way I liked your decomposition of the r2 in your other post:)
Dec 19, 2021 at 10:05 comment added janrth The scaler only denominator has specific properties for purely linear functions. For example it will be between 0 and 1 for linear functions. But it can be different for non-linear relations. To be honest I have never went deep into the theoretical explanation. But it is what you find pretty often when people talk about r2 and also how I understand r2. Try to compare a linear function against a concave function. I think what will happen is that the denominator will behave differently. Play around with different prediction values and try to compare r2 against the mse for both functions.
Dec 18, 2021 at 22:53 comment added Dave $R^2$ is a monotonic function of MSE. $$R^2=1-\dfrac{nMSE}{\sum_i(y_i-\bar y)^2}$$
Dec 18, 2021 at 21:11 comment added janrth There is this paper everybody is referring to. They did tons of simulations and found how r2 is invalid in picking the best model. Mse and mae should always work no matter how non-linear your data is. Then only the question of outliers would decide if you go with mse or mae. here is the paper: Spiess, Andrej-Nikolai, Natalie Neumeyer. An evaluation of R2 as an inadequate measure for nonlinear models in pharmacological and biochemical research: a Monte Carlo approach. BMC Pharmacology. 2010; 10: 6.
Dec 18, 2021 at 12:50 comment added Dave Minimizing (r)mse is equivalent to maximizing the “invalid” $R^2$.
Dec 18, 2021 at 11:05 comment added janrth good question. Probably (r)mse. But I like to also look at mae, because it is very straight forward in the interpretation.
Dec 17, 2021 at 11:54 comment added Dave How would you pick the best tree-based regression model, mean squared error or RMSE?
Dec 17, 2021 at 11:44 comment added janrth But isn't it the case that the sum of residuals for a non-linear situation do not behave to the mean prediction as it should be in a linear case, which makes the interpretation basically non-valid. This is what I mean by mathematically invalid, because the metric does not return what you want to see. I would for example not pick the best model based on r2 for a tree based algorithm. But anyway....:) It is still used everywhere:)
Dec 16, 2021 at 21:06 comment added Dave $R^2$ is funky in the nonlinear case, but "not...mathematically valid for non-linear regressions" takes it too far. $R^2$ is just as valid as $MSE$.
Dec 16, 2021 at 20:52 history answered janrth CC BY-SA 4.0