8
$\begingroup$

Why in literature usually the common accuracy measures like MAD, MSE, RMSE, MAPE ... are used. Why not using the $R^2$ (coefficient of determination)?

I was thinking about the difference: By using the MSE i can compare the average of the forecast. And when using $R^2$ I'll get a information about the variance.

Why the comparison of averages is mostly commonly used? Can anybody give me an hint?

$\endgroup$
7
$\begingroup$

In-sample $R^2$ is not a suitable measure of forecast accuracy because it does not account for overfitting. It is always possible to build a complicated model that will fit the data perfectly in sample but there are no guarantees such a model would perform decently out of sample.


Out-of-sample $R^2$, i.e. the squared correlation between the forecasts and the actual values, is deficient in that it does not account for bias in forecasts.

For example, consider realized values

$$y_{t+1},\dotsc,y_{t+m}$$

and two competing forecasts:

$$\hat{y}_{t+1},\dotsc,\hat{y}_{t+m}$$

and

$$\tilde{y}_{t+1},\dotsc,\tilde{y}_{t+m}.$$

Now assume that

$$\tilde{y}_{t+i}=c+\hat{y}_{t+i}$$

for every $i$, where $c$ is a constant. That is, the forecasts are the same except that the second one is higher by $c$. These two forecasts will generally have different MSE, MAPE etc. but the $R^2$ will be the same.

Consider an extreme case: the first forecast is perfect, i.e. $\hat{y}_{t+i}=y_{t+i}$ for every $i$. The $R^2$ of this forecast will be 1 (which is very good). However, the $R^2$ of the other forecast will also be 1 even though the forecast is biased by $c$ for every $i$.

$\endgroup$
  • $\begingroup$ Why would you calculate the out-of-sample $R^2$ as the squared correlation instead of an out-of-sample $1-\frac{SSE}{TSS}$? Doing an R simulation, I get your same result that $R^2$ is unchanged upon biasing the out-of-sample predictions if I calculate $R^2$ your way, but calculating $R^2$ my way gives a terrible out-of-sample value (negative, even). $\endgroup$ – Dave Dec 30 '19 at 19:00
  • $\begingroup$ @Dave, thank you for your insight. I do not see it as my way vs. someone else's way, rather as a direct extension of a definition of an in-sample $R^2$. After all, the letter R denotes correlation, and the in-sample $R^2$ is the square of the multiple correlation between the actual and the fitted values. There are other possible ways of defining in-sample $R^2$, leading to the same in-sample quantity. Meanwhile, their out-of-sample extensions are not equivalent. My answer shows that a direct extension (though not all possible ones) can be nonsensical as a measure of forecast accuracy. $\endgroup$ – Richard Hardy Dec 30 '19 at 21:07
  • $\begingroup$ @Dave, in any case, yours is a good point! I could add a note saying that not all extensions of the in-sample $R^2$ to the out-of-sample case are so ill behaved. I would have to think which of them are sensible as measures of forecast accuracy and which of them are not. $\endgroup$ – Richard Hardy Dec 30 '19 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.