0
$\begingroup$

Pardon me, I am new to timeseries forecasting. Given that there is not always a clear cut way to know whether your forecasting model is good enough and there's a significant degree of subjectivity in measuring this or even defining what "good enough" means, I thought it would be interesting and educative to find out what people do in practice.

What are the modelling / quantitative criteria that you use to determine that you have a good enough timeseries forecasting model in practice?

I define a model that's good enough as one that produces reasonable enough forecasts of a timeseries in practise. Perhaps the question should be: what are the modelling/quantitative criteria that you use to determine that you have a model whose forecasts you believe to be reasonable? Are there certain things you would not accept for your forecasting model (e.g. correlated residuals) - what are they and why?

(You may assume that you have a good idea of what the regressors are and you have the future values for them)

$\endgroup$
16
0
$\begingroup$

If you are using R, you can use the predict function (https://stat.ethz.ch/R-manual/R-devel/library/stats/html/predict.lm.html) to compare your model's predicted values to the actual values.

Of course, if your model is designed as a forecasting tool, you may not be able to assess the future "goodness of fit" currently, but you should still be able to apply the predict function to data where the values of the response variable are known.

Other common measures of fit include RMSE, R-squared, and MAE, all of which can be drawn from the postResample function in caret. Link here: https://www.rdocumentation.org/packages/caret/versions/2.27/topics/postResample.

As you mentioned, autocorrelation is another problem to consider in evaluating time series models. You can use the acf function to quantify and visualize autocorrelation (https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/acf). Corrections to autocorrelation include robust standard errors, and the inclusion of lag terms.

$\endgroup$
0
$\begingroup$

I do this for a living and so its important to me.:) However, I am a data analyst rather than a statistician so my answer might vary from a statistician's. The way I assess it if my model is good enough is two fold. First, I track a percent difference in each month and a year to date (the year is what really matters to us). My rule of thumb is that five percent error is acceptable given our uncertain process, but each person has to make that decision themselves. I don't think there is an objective way to decide that which is widely accepted and it also depends on how certain and unchanging your process is. Second, I have tried to find (it is not easy) what the rate of error is for others in my area. That gives me a benchmark to compare my results to.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.