Regression: What is the utility of $R^2$ compared with RMSE?

Question

Suppose I'm doing regression with training, validation, and test sets. I can find RMSE and R squared ($R^2$, the coefficient of determination) from the output of my software (such as R's lm() function).

My understanding is that the test RMSE (or MSE) is the measure of goodness of predicting the validation/test values, while $R^2$ is a measure of goodness of fit in capturing the variance in the training set.

In the real world, what I really care about is generalized prediction accuracy on data I haven't seen. So then what is the utility of the $R^2$ value compared with RMSE?

Answer 1

The unadjusted $R^2$ is defined to be $$R^2 = 1 - \frac{\frac{1}{n}\sum_{i=1}^n (y_i - \hat y_i)^2}{\frac{1}{n}\sum_{i=1}^n (y_i - \bar y)^2} = 1 - \frac{MSE}{\frac{1}{n}TotSS}$$

Let's take the RMSE to be $$ RMSE = \sqrt{MSE}. $$

For a given data set $y_i$ and $\bar y$ are fixed, so as different models are considered only the $\hat y_i$ change. This means that in the above expressions, only the MSE changes. So both $R^2$ and $RMSE$ are functions of the same thing, and therefore there isn't much of a difference (except for interpretation) by considering one versus the other.

If we instead look at the adjusted $R^2$ or use $RMSE = \sqrt{\frac{n}{n-p}MSE}$ then we'll also have $p$, the dimension of the model, changing for different models.

Answer 2

Chaconne did an excellent job about defining the measures formulas and how they are very closely related from a math standpoint. If you benchmark or rank models using the same data set those two measures are interchangeable, meaning you will get the exact same ranking of your models whether you use R Square (ranking them high to low) or the RMSE (ranking them low to high).

However, the two measures have a very different meaning and use. R Square is not only a measure of Goodness-of-fit, it is also a measure of how much the model (the set of independent variables you selected) explain the behavior (or the variance) of your dependent variable. So, if your model has an R Square of 0.60, it explains 60% of the behavior of your dependent variable. Now, if you use the Adjusted R Square that essentially penalizes the R Square for the number of variables you use you get a pretty good idea when you should stop adding variables to your model (and eventually just get a model that is overfit). If your Adjusted R Square is 0.60. And, when you add an extra variable it just increases to 0.61. It is probably not worth it adding this extra variable.

Now, turning to RMSE also most commonly referred to as a Standard Error. It has a completely different use than R Square. The Standard Error allows you to build Confidence Intervals around your regression estimate assuming whatever Confidence Level you are interested in (typically 99%, 95%, or 90%). Indeed, the Standard Error is the equivalent of a Z value. So, if you want to build a 95% CI around your regression trendline you multiply the Standard Error by 1.96 and quickly generates a high and low estimate as border of your 95% CI around the regression line.

So, both R Square (and Adjusted R Square) and the Standard Error are extremely useful in assessing the statistical robustness of a model. And, as indicated they have completely different practical application. One measures the explanatory power of the model. The other one allows you to build Confidence Intervals. Both, very useful but different stuff.

Regarding assessing prediction accuracy on data you have not seen, both measures have their limitations as well as most other measures you can think off. On new data that is out-of-sample, the R Square and Standard Error on the history or learning sample of the model will not be of much use. The out-of-sample stuff is just a great test to check whether your model is overfit (great R Square and low Standard Error, but poor performance in out-of-sample) or not. I understand better measures for prospective data (data you have not seen yet) are the information criterion including AIC, BIC, SIC. And, the model with the best information criterion values should handle unseen data better, in other words be more predictive. Those measures are close cousins of the Adjusted R Square concept. However, they are more punitive on adding additional variables than Adjusted R Square is.

Answer 3

$R^2$ forces you to compare model performance to the performance of a baseline model.

$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) $$

The numerator is a function of the $\text{RMSE}$ of your model (square it and the multiply by the sample size, $N$). The denominator is that same function of the $\text{RMSE}$ of a model that always predicts $\bar y$, which is a reasonable baseline to which performance can be compared: if you want to predict the conditional mean and have no idea how to do that, what better option than predicting the marginal/pooled mean $\bar y$ every time?

$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) =1 - \left(\dfrac{ N\times \left(\text{RMSE}_{\text{model}}\right)^2 }{ N\times \left(\text{RMSE}_{\bar y}\right)^2 }\right) =1 - \left(\dfrac{ \left(\text{RMSE}_{\text{model}}\right)^2 }{ \left(\text{RMSE}_{\bar y}\right)^2 }\right) $$

Since $R^2$ forces you to compare to a benchmark, you avoid making silly claims just because the $\text{RMSE}$ appears to be a small number. Sure, the number might be small, but if you would have an even smaller (better) $\text{RMSE}$ using a basic model, you probably want to know if all of the hard work you've put in to develop your model has resulted in worse performance than if you just predicted the same $\bar y$ every time.

Yes, you will get similar information by just looking at the model $\text{RMSE}$ and comparing it to the $\text{RMSE}$ of a model that always predicts $\bar y$, but calculating $R^2$ explicitly forces you to do this.

A drawback of $R^2$ is that it is easy to get into a trap of looking at values like letter grades in school, where $R^2 = 0.95$ is an $\text{A}$ that makes you happy and $R^2 = 0.50$ is an $\text{F}$ that makes you sad. If the state-of-the-art in modeling only achieves $R^2 = 0.30$, then your $R^2 = 0.50$ doesn't sound so bad, and if people are routinely scoring $R^2 > 0.99$, then your $R^2 = 0.95$ doesn't sound so great. Out of context, it is difficult to consider model performance as good or bad, yet $R^2$ can give the illusion of aligning with letter grades.