7
$\begingroup$

Why in literature usually the common accuracy measures like MAD, MSE, RMSE, MAPE ... are used. Why not using the $R^2$ (coefficient of determination)?

I was thinking about the difference: By using the MSE i can compare the average of the forecast. And when using $R^2$ I'll get a information about the variance.

Why the comparison of averages is mostly commonly used? Can anybody give me an hint?

$\endgroup$
7
$\begingroup$

In-sample $R^2$ is not a suitable measure of forecast accuracy because it does not account for overfitting. It is always possible to build a complicated model that will fit the data perfectly in sample but there are no guarantees such a model would perform decently out of sample.


Out-of-sample $R^2$, i.e. the squared correlation between the forecasts and the actual values, is deficient in that it does not account for bias in forecasts.

For example, consider realized values

$$y_{t+1},\dotsc,y_{t+m}$$

and two competing forecasts:

$$\hat{y}_{t+1},\dotsc,\hat{y}_{t+m}$$

and

$$\tilde{y}_{t+1},\dotsc,\tilde{y}_{t+m}.$$

Now assume that

$$\tilde{y}_{t+i}=c+\hat{y}_{t+i}$$

for every $i$, where $c$ is a constant. That is, the forecasts are the same except that the second one is higher by $c$. These two forecasts will generally have different MSE, MAPE etc. but the $R^2$ will be the same.

Consider an extreme case: the first forecast is perfect, i.e. $\hat{y}_{t+i}=y_{t+i}$ for every $i$. The $R^2$ of this forecast will be 1 (which is very good). However, the $R^2$ of the other forecast will also be 1 even though the forecast is biased by $c$ for every $i$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.