# Regression: What is the utility of R squared compared to RMSE?

Suppose I'm doing regression with training, validation, and test sets. I can find RMSE and R squared (R^2, the coefficient of determination) from the output of my software (such as R's lm() function).

My understanding is that the test RMSE (or MSE) is the measure of goodness of predicting the validation/test values, while R^2 is a measure of goodness of fit in capturing the variance in the training set.

In the real world, what I really care about is generalized prediction accuracy on data I haven't seen. So then what is the utility of the R^2 value compared to RMSE?

The unadjusted $R^2$ is defined to be $$R^2 = 1 - \frac{\frac{1}{n}\sum_{i=1}^n (y_i - \hat y_i)^2}{\frac{1}{n}\sum_{i=1}^n (y_i - \bar y)^2} = 1 - \frac{MSE}{\frac{1}{n}TotSS}$$

Let's take the RMSE to be $$RMSE = \sqrt{MSE}.$$

For a given data set $y_i$ and $\bar y$ are fixed, so as different models are considered only the $\hat y_i$ change. This means that in the above expressions, only the MSE changes. So both $R^2$ and $RMSE$ are functions of the same thing, and therefore there isn't much of a difference (except for interpretation) by considering one versus the other.

If we instead look at the adjusted $R^2$ or use $RMSE = \sqrt{\frac{n}{n-p}MSE}$ then we'll also have $p$, the dimension of the model, changing for different models.

• I've always thought that if you have high RMSE your model is not reliable therefore other metrics that explain the model is also not reliable. But you are saying no matter how high RMSE is if one model1 has higher R**2 than model2 it implies model1 is better? Oct 12, 2021 at 7:13
• @haneulkim RMSE has the same units as the response so what counts as large or small depends on the context; for example, whether or not an RMSE of 100 is "big" isn't something that can be answered without knowing more about the problem. $R^2$ essentially normalizes the MSE into a unitless quantity so that we can interpret it context-free. If two models use the same response, so the total sum of squares is the same, then comparing $R^2$s is equivalent to comparing RMSEs. Does that help?
– jld
Oct 13, 2021 at 16:37
• Yes, thank you! One more question though. If you have poor regression model(Large errors) with multiple x-variables is it valid to conclude x1 is more important than x2 if $R^2$ of x1 is greater? Oct 14, 2021 at 2:18
• @haneulkim I'm not sure if I follow. What are x1 and x2? Do you mean like saying one model is better than another due to higher $R^2$, despite there being lots of noise?
– jld
Oct 14, 2021 at 3:09
• @haneulkim that's a good question, I think you should just ask that on CV!
– jld
Oct 19, 2021 at 17:22

Chaconne did an excellent job about defining the measures formulas and how they are very closely related from a math standpoint. If you benchmark or rank models using the same data set those two measures are interchangeable, meaning you will get the exact same ranking of your models whether you use R Square (ranking them high to low) or the RMSE (ranking them low to high).

However, the two measures have a very different meaning and use. R Square is not only a measure of Goodness-of-fit, it is also a measure of how much the model (the set of independent variables you selected) explain the behavior (or the variance) of your dependent variable. So, if your model has an R Square of 0.60, it explains 60% of the behavior of your dependent variable. Now, if you use the Adjusted R Square that essentially penalizes the R Square for the number of variables you use you get a pretty good idea when you should stop adding variables to your model (and eventually just get a model that is overfit). If your Adjusted R Square is 0.60. And, when you add an extra variable it just increases to 0.61. It is probably not worth it adding this extra variable.

Now, turning to RMSE also most commonly referred to as a Standard Error. It has a completely different use than R Square. The Standard Error allows you to build Confidence Intervals around your regression estimate assuming whatever Confidence Level you are interested in (typically 99%, 95%, or 90%). Indeed, the Standard Error is the equivalent of a Z value. So, if you want to build a 95% CI around your regression trendline you multiply the Standard Error by 1.96 and quickly generates a high and low estimate as border of your 95% CI around the regression line.

So, both R Square (and Adjusted R Square) and the Standard Error are extremely useful in assessing the statistical robustness of a model. And, as indicated they have completely different practical application. One measures the explanatory power of the model. The other one allows you to build Confidence Intervals. Both, very useful but different stuff.

Regarding assessing prediction accuracy on data you have not seen, both measures have their limitations as well as most other measures you can think off. On new data that is out-of-sample, the R Square and Standard Error on the history or learning sample of the model will not be of much use. The out-of-sample stuff is just a great test to check whether your model is overfit (great R Square and low Standard Error, but poor performance in out-of-sample) or not. I understand better measures for prospective data (data you have not seen yet) are the information criterion including AIC, BIC, SIC. And, the model with the best information criterion values should handle unseen data better, in other words be more predictive. Those measures are close cousins of the Adjusted R Square concept. However, they are more punitive on adding additional variables than Adjusted R Square is.

• Thanks for your answer. I have typically used RMSE only for assessing the predictive power of a linear regression model (after predicting the values of an unseen test set). So I've not seen that RMSE "has a completely different use ... to build confidence intervals around your regression estimate." I guess this must be a statistician thing? I'm from computer science, so I haven't computed very many confidence intervals in my career. May 28, 2016 at 18:42