Accounting for uncertainty in evaluating a forecast

Question

I'm new to forecasting and I have some (probably very) basic and generic questions. I'd appreciate some references that get into details of this too.

Using some model to forecast a time series, I generate a number of MCMC samples and use the median as the forecast and an interval around it as the prediction interval. My question is:

1. What are some standard practices in choosing the interval width. in particular, why not 100%?
2. When would one say the fitted model is NOT a good fit? i.e. looking at a particular measure of goodness-of-fit, like $R^2$ or normalized root mean square error, how does one choose a threshold for accepting/rejecting the model?
3. Are there any goodness-of-fit measures that (somehow) take into account the already measured uncertainties?

Note 1: for practical reasons I'm not interested in out-of-sample validations.

Note 2: I'm probably asking "wrong" question, please feel free to direct me to the "right" questions I should be asking.

Answer 1

1. Your choice of Prediction Interval (PI) width depends on what you plan on doing with the PI. For instance, if you will use it to set safety stocks, you can determine the optimal quantile, be it 80%, 90% or 95%. (In this case, you would usually not use both endpoints of the PI, but only the upper one.)

   Using a "100% PI" often does not make a lot of sense. On the one hand, it's a function of how many MCMC samples you draw, because the maximum and the minimum can only get more extreme as you collect more samples, but the user will usually have a problem accepting that if you run your data collection longer, you get a higher and higher maximum. Also, the true future distribution may be unbounded (at least in theory), but your "100% PI" will always be bounded, so it will be a poor approximation to infinity.

2. First, don't evaluate your model in-sample. You will always overfit. Instead, use a holdout sample and evaluate forecast accuracy on that. Yes, even if you are not interested in this per se. Just hold out some of your training data.

   Then you can accept your model and forecast if it is good enough for your application and you can't improve it any more with reasonable effort. Yes, this is subjective. How to know that your machine learning problem is hopeless?

3. I'm not quite sure what you mean by "already measured uncertainties". You can use proper scoring rules to assess full predictive distributions, which your MCMC samples will give you. This is very good practice. I recommend Gneiting & Katzfuss (2014).

   Alternatively, you can evaluate your PIs, but this loses a lot of information. One quality measure is the Mean Scaled Interval Score (Gneiting & Raftery, 2007), which was used in the recent M4 forecasting competition (Makridakis et al., 2020](https://doi.org/10.1016/j.ijforecast.2019.04.014)).

I very much recommend Forecasting: Principles and Practice (2nd ed.) by Athanasopoulos & Hyndman.

Answer 2

I would first try some basic tests to ascertain if your methodology is producing any forecast of value.

For example, assume that there is a possible interest in a correct forecast greater (or less) than say k%. Tabulate the number of times the database indicated an expected forecast value greater than k% in total of n forecast cases. Compare this to the actually observed times it correctly occurred. Use a statistical test to assign significance (like, for example, a Pearson's Chi-Square test).

Repeat for different potential values of interest and record statistical test results.

How is your model performing? You now should be better able to answer the question: "When would one say the fitted model is NOT a good fit?" Also, you now have an intuitive "goodness-of-fit measures that (somehow) take into account the already measured uncertainties".

Answer 3

I think ultimately how good your model is, is how well you predict the future (out of sample predictions). There is no "correct" answer to that, ultimately it is what you and/or those you report will accept. Its worth considering how well others do relative to your predictions. In some areas with high process uncertainty its difficult to be consistently right. Reviews I have seen suggest uncertainty is consistently under estimated when predicting future values. Some have suggested ways to correct for this, they are not tied to formal academic or mathematical theory (at least those I have seen). As the COVID19 crisis reminds us structural breaks or pulses are pretty much impossible to predict and wreck the best models. Even if you predict well in the past, the process may change and then you will not predict correctly until you adjust your model.

Answer 4

A little information regarding your first point on the selection of Prediction Intervals.

In general, forecasts are used as an input in a subsequent decision making process, usually an optimization problem. For example, given a set of stock price predictions, maximize portfolio returns. In that regard, it is common to make decisions that take risk under consideration, which in that case is the uncertainty in stock price predictions. In some cases, the risk is only considered on one side of the distribution (in the previous example negative returns). From my experience usual values are 1%, 5% or 10% which correspond to 98%, 90% and 80% Prediction Intervals, if you consider both sides of the distribution.

Of course, these values depend on the problem at hand and the specific domain of application. But I would not suggest estimating Prediction Intervals over 98%, as you have to be absolutely sure about the distribution at hand.