Is it appropriate when forecasting to use $R^2$ as the measure of how well exponential smoothing fits a data set for the purpose of time-series forecasting?

I understand that it is appropriate for regression analysis; could I compare $R^2$ values between a linear regression, logarithmic regression and exponential smoothing as a method of evaluating the best technique for forecasting?


In-sample fit such as $R^2$ is even more frowned upon as a measure of model quality in forecasting than in other statistical subdisciplines, for all the well-known reasons (if you make your model more and more complex, you will get better and better in-sample fits... but ever worse out-of-sample forecast accuracy). If at all, people will rather use information criteria, such as AIC or BIC.

However, the gold standard in forecast model assessment and model selection is using a hold-out sample. Suppose you have $n$ historical data points. Remove the last $k$ (say, $k=10$, depending on your data and the forecast horizon you actually are interested in). Fit your model (exponential smoothing, whatever) on the first $n-k$ observations, use the fitted model to forecast into the last $k$ observations, and compare this forecast with the hold-out actuals. Do this for a couple of models and choose the one with the best performance on the hold-out. Finally, refit this model using all $n$ observations to create your actual forecast.

There are many, many different error measures you can use in the comparison step, and they will not all yield the same result. Look at multiple measures and see whether the results are roughly consistent. This page explains some forecast error measures. And the entire free open source forecasting textbook is an extremely good source of information for forecasting.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.