4
$\begingroup$

Consider a variable $Y$ (e.g., temperature). Suppose that we were able to estimate this variable each year for the past $N$ years using some type of model. This means we have access to annual estimated values for Y (denoted as $Y_1, \ldots, Y_N$) and associated standard errors $S_1,\ldots,S_N$. The goal is to produce a point forecast for the value of $Y$ at the time N+1 which incorporates the uncertainty present in the estimated annual values $Y_1,\ldots,Y_N$.

One way to proceed is to use Monte Carlo simulation to create an ensemble of $B$=100,000 (or a large enough number) of synthetic time series obtained by shifting each of the original values $Y_1,\ldots,Y_N$ by a random z score (i.e., Gaussian white noise), scaled by the standard error $S_t$. (This approach assumes that $Y_1,\ldots,Y_N$ are independent.)

Each synthetic series can then be used to produce (I) a point forecast of $Y$ at time $N+1$ and (II) an interval forecast of $Y$ at time $N+1$.

My question is:

How do we summarize the information conveyed by the simulated point forecasts and interval forecasts to quantify the uncertainty present in $Y_1,\ldots,Y_N$ and its impact on the forecasting output?

For point forecasts, it makes sense to report the simulation distribution of the point forecasts. But what aspect of this distribution captures the uncertainty (e.g., spread)?

For interval forecasts, it is not clear (at least not to me) how to proceed. Is there any way to quantify uncertainty in the forecasting input (i.e., $Y_1,\ldots,Y_N$) when it comes to width and/or coverage of these intervals? (Maybe by using some type of retrospective performance of the prediction procedure?)

$\endgroup$
3
$\begingroup$

You have two sources of uncertainty: the uncertainty in the historical data, and the uncertainty in producing the forecasts given the historical data. The simulation distribution of point forecasts is capturing the uncertainty in the historical data only.

To capture the joint uncertainty, I suggest you simulate a future value from the forecast distribution for each of the synthetic time series. That is, for each synthetic time series compute the point forecast and the forecast variance, and then simulate a value from this distribution. These simulated future values then include the uncertainty in both the forecast distribution and in the historical data.

You could compute a prediction interval from the percentiles of the future values and compare its width with the size of the prediction interval produced for each synthetic series.

$\endgroup$
  • $\begingroup$ Thank you for your excellent answer, Rob. Still wondering:Can the forecast distribution be a bootstrap distribution if forecast variance is not readily available (even though this is more expensive computationally)? Can a typical value from the distribution of future values (e.g., mean/median) be used as the "final" point forecast, which will incorporate the two sources of uncertainty (historical data + producing the forecasts). $\endgroup$ – Isabella Ghement Feb 15 '14 at 18:35
  • $\begingroup$ A typical value is not the mean or median. I said to simulate a future value. Then you are taking into account the forecast variance. $\endgroup$ – Rob Hyndman Feb 15 '14 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.