2
$\begingroup$

I understand the general idea of different time series model fittings, calculations, and model comparison.

However, I am a little confused of understanding the forecast of a time series model. For example, if my model predictions for a forecast horizon such as h=14 are all above (or lower) than the actual value, does it necessarily mean the model is bad?

I asked because maybe the model are able to capture all the patterns except there's a shift (maybe due to a drift / intercept term), causing the prediction plot is almost parallel to the actual data, except there's a gap in between.

I guess my questions can be summarized in:

1) How does one determined if the model forecasts are good ? For now I am using MAE, Mean, CV for Error to check on the forecast.

2) Consider two models with predictions having same MAE, in which one has all above (or below) predictions and the other having some higher and lower predictions, in a general sense, which one of them would be considered a better forecast?

$\endgroup$
3
$\begingroup$

The answer to both questions really depends on the answer to a third: What are you using the forecast for?

The answer to this question is hugely informative toward evaluating the forecast, as it allows you to answer subsidiary questions such as "Which has a larger negative effect, under-forecasted values or over-forecasted values?" and "Is it more important to capture the long-term trend (approximate underlying function) or to detect observations which may strongly deviate from that function?"

The metrics you mention may not be totally sufficient to allow you to evaluate comparative models in the context of those answers, but in many cases they would be.

The important thing to realize is that the application of the model actually controls the answer to how to evaluate the model itself.

In effect, the context of the forecast is not divorceable from its application and there is no controlling general case (except perhaps in certain subject areas, where under- or over- forecasting may be seen as preferable by general agreement, another form of context).

Practical example, from the comments:

In a sales forcasting situation, where under-forecasted sales might result in missed opportunities (from under-staffing) which might be more costly than additional, unnecessary labor cost (from over-staffing), you might prefer over-forecasting to under-forecasting.

If, in the same context, your forecast perfectly predict sales volume on all but the three heaviest sales days of the year, where it spikes to quadruple normal values, you might find that the fact that you capture the rest of the time perfectly doesn't save your job in spite of a low overall MSE.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your reply. Are you saying the forecast "efficacy" should be based on the goal? Let say I am forecasting sales next week, over-forecasted values might result in better staffing of the store so its preferred over under-forecasted values. In the case of providing an general prediction of sales, a model with fluctuating predictions would result in a better average prediction. Is that right? $\endgroup$ – Matthew Lau Dec 23 '15 at 18:30
  • $\begingroup$ Exactly- in the case where underforecasted sales might result in missed opportunities (from under-staffing) which might be more costly than additional, unnecessary labor cost (from over-staffing), you might prefer over-forecasting to under-forecasting. $\endgroup$ – Thomas Cleberg Dec 23 '15 at 18:34
  • $\begingroup$ If, in the same context, your forecast perfectly predict sales volume on all but the three heaviest sales days of the year, where it spikes to quadruple normal values, you might find that the fact that you capture the rest of the time perfectly doesn't save your job in spite of a low MSE. $\endgroup$ – Thomas Cleberg Dec 23 '15 at 18:52
  • $\begingroup$ Thanks a lot ! I will twist with my model and determine which one of them provides the "best" forecast. $\endgroup$ – Matthew Lau Dec 23 '15 at 19:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.