11
$\begingroup$

Why in literature usually the common accuracy measures like MAD, MSE, RMSE, MAPE ... are used. Why not use the $R^2$ (coefficient of determination)?

I was thinking about the difference: By using the MSE I can compare the average of the forecast. And when using $R^2$ I'll get a information about the variance.

Why is the comparison of averages used most commonly? Can anybody give me an hint?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
11
$\begingroup$

In-sample $R^2$ is not a suitable measure of forecast accuracy because it does not account for overfitting. It is always possible to build a flexible model that will fit the data perfectly in sample but there are no guarantees such a model would perform decently out of sample.

Out-of-sample $R^2$, i.e. the squared correlation between the forecasts and the actual values, is deficient in that it does not account for bias in forecasts.

For example, consider realized values

$$y_{t+1},\dotsc,y_{t+m}$$

and two competing forecasts:

$$\hat{y}_{t+1},\dotsc,\hat{y}_{t+m}$$

and

$$\tilde{y}_{t+1},\dotsc,\tilde{y}_{t+m}.$$

Now assume that

$$\tilde{y}_{t+i}=c+\hat{y}_{t+i}$$

for every $i$, where $c$ is a constant. That is, the forecasts are the same except that the second one is higher by $c$. These two forecasts will generally have different MSE, MAPE etc. but the $R^2$ will be the same.

Consider an extreme case: the first forecast is perfect, i.e. $\hat{y}_{t+i}=y_{t+i}$ for every $i$. The $R^2$ of this forecast will be 1 (which is very good). However, the $R^2$ of the other forecast will also be 1 even though the forecast is biased by $c$ for every $i$.

Improve this answer
$\endgroup$
7
  • 3
    $\begingroup$ Why would you calculate the out-of-sample $R^2$ as the squared correlation instead of an out-of-sample $1-\frac{SSE}{TSS}$? Doing an R simulation, I get your same result that $R^2$ is unchanged upon biasing the out-of-sample predictions if I calculate $R^2$ your way, but calculating $R^2$ my way gives a terrible out-of-sample value (negative, even). $\endgroup$
    – Dave
    Commented Dec 30, 2019 at 19:00
  • 1
    $\begingroup$ @Dave, thank you for your insight. I do not see it as my way vs. someone else's way, rather as a direct extension of a definition of an in-sample $R^2$. After all, the letter R denotes correlation, and the in-sample $R^2$ is the square of the multiple correlation between the actual and the fitted values. There are other possible ways of defining in-sample $R^2$, leading to the same in-sample quantity. Meanwhile, their out-of-sample extensions are not equivalent. My answer shows that a direct extension (though not all possible ones) can be nonsensical as a measure of forecast accuracy. $\endgroup$
    – Richard Hardy
    Commented Dec 30, 2019 at 21:07
  • 1
    $\begingroup$ @Dave, in any case, yours is a good point! I could add a note saying that not all extensions of the in-sample $R^2$ to the out-of-sample case are so ill behaved. I would have to think which of them are sensible as measures of forecast accuracy and which of them are not. $\endgroup$
    – Richard Hardy
    Commented Dec 30, 2019 at 21:10
  • $\begingroup$ A few years later, I’ve posted an answer that is related to our discussion. After thinking about it, it’s not at all clear what $TSS$ should be (though I believe the field would benefit from considering what would make sense). $\endgroup$
    – Dave
    Commented Feb 21, 2023 at 13:12
  • $\begingroup$ @Dave, this reminds me of Theil's $U$ statistic. The funny thing is, there has been confusion as to what that denotes, too... $\endgroup$
    – Richard Hardy
    Commented Feb 21, 2023 at 14:04
1
$\begingroup$

It’s not clear how $R^2$ should be defined in such a scenario.

In another answer, Richard Hardy accurately points out the issues of squaring the correlation between the predicted and actual values. I give some graphs of that kind of problem in my answer to a related question. Consequently, such a definition of $R^2$ leads to a calculated value that is less helpful than one might hope.

Then there’s my idea to compare the square loss of your model to the square loss of a baseline “must beat” model. However, it is not clear what such a baseline should be in time series forecasting. Do you use the mean of some subset of the data? Do you use the mean of all periods before your forecast? Do you use mean of all true observations, even though you would not have had access to all of those observations when you had to make your predictions? That is how Python's sklearn.metrics.r2_score would do it. For financial predictions, I could see using a historical model of some index, such as knowing the historical return of the S&P 500 or (probably even better) the return on the S&P 500 over the same period.

Because of this ambiguity in how to define $R^2$ and what would make for a useful calculation, such a metric seems to be avoided: the obvious calculation based on correlation has major issues, and it is not clear what remedy is appropriate.

I do believe there would be value to making such a comparison to some kind of baseline model, however. For instance, a financial advisor might boast to clients about making them a $15\%$ return on their money. That sounds impressive, but if the clients could have invested in the S&P 500 over that same time and made $17\%$, the clients should be disappointed with their advisor.

(There are all kinds of complexities when it comes to a real financial problem, such as older people near retirement not wanting to incur the risks that go along with stock investing, but I think this illustrates why some kind of comparison to a baseline model would be valuable. (An additional complication could be fees paid to an advisor or money manager.))

Improve this answer
$\endgroup$
1
  • $\begingroup$ It is easy to calculate $R^2$ once it has been defined. I would consider rephrasing the first sentence accordingly. $\endgroup$
    – Richard Hardy
    Commented Feb 21, 2023 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.