Deriving risk estimates using forecasting confidence limits and out of sample hold-out cases

Question

I was hoping for some advice. I use SAS for automatic forecasting (I have a large number of forecasts to complete in a limited timeframe).

As part of the forecast output from SAS, I get a mid-point (median or mean), and an upper and lower confidence limit for each forecast. This is determined at a pre-specified level (i.e. 95%). Evidently, the confidence limits are derived statistically. I appreciate that within the upper and lower limits there is a range of potential forecasting results that could occur.

Based on my interpretation, values close to the upper and lower limits of the forecasts will be less likely because they depart a fair way from the mean/median and are close to the tails in terms of the distribution of possible forecasted values.

In a number of potential forecasting scenarios I face each month, I have a target the business needs to achieve by the end of the next financial year (june 30, 2012). I also have a forecast of the likely future value + upper and lower forecast limits for end of year (june 30, 2012) derived using moderate to long term historical performance. I need to quantify the probability of reaching a target given a forecasted result, using the point where the target is located relative to the mean/median and also the upper and lower forecast limits. For example, if the series has a mean forecast of 50, a LCL of 0 and UCL of 100 and the target is 75, I need to quantify the probability this target will be met at June 30, 2012.

It seems to me I can use the fact the target of 75 falls on the 75th percentile relative to the forecast upper and lower forecast limits.

• Is it fair to say that there is a 25% chance of hitting the target under an assumed uniform distribution?
• Or is it more appropriate to assume the forecast series within the upper and lower limits is normally distributed with a greater concentration of values in the CDF closer to the mean?
• Or is this model dependent (i.e. different for arima and esm models)

Also, as a rule of thumb how many holdouts do people use for

1. a series with 12 months (I am using 10% or 1)
2. a series with 24 months (I am using 10% or 2)

Answer 1

In theory the confidence intervals are derived from the estimated distribution of the forecast. This means that the model gives estimate $\hat{y}$ and its estimated cdf $\hat{F}$. The 95%-confidence interval is then calculated as

$$(\hat{F}^{-1}(0.025),\hat{F}^{-1}(0.975))$$

If $\hat{F}$ would be available you could assign (with certain assumptions) probability for the target $y$ by simply reporting $1-\hat{F}(y)$ which will be the probability $P(\hat{y}>y)$, i.e. the probability that the forecast will be larger than the target.

However you do not have $\hat{F}$, you just have 2 values: $\hat{F}^{-1}(0.025)$ and $\hat{F}^{-1}(0.975)$. In general this is too little information to recover $\hat{F}$. On the other hand it helps to know that for a lot of models $\hat{F}$ is normal distribution. Usually:

$$\hat{y}\sim N(\mu_{\hat y},\sigma^2)$$

where $\mu_{\hat y}$ is the reported mean or median forecast. Since in this case the 95% confidence interval is

$$(\mu_{\hat y}-1.96\sigma,\mu_{\hat y}+1.96\sigma)$$

It is not hard then to recover $\sigma$ and for the target $y$ we have

$$P(\hat{y}>y)=1-\Phi\left(\frac{y-\mu_{\hat y}}{\sigma}\right)$$

where $\Phi$ is standard normal distribution.

This should be used with caution, since although a lot of models do use normal distribution for $\hat{F}$, this is not always the case. Furthermore it might be that for example $\log y$ was modelled, but confidence interval was reported for $y$.

So to sum up the answers to your questions would be:

1. You can say that, but the uniform distribution assumption is almost guaranteed to be false.
2. Yes you can suppose that with some confidence, but you need to check the model details to be sure.
3. Yes it is model dependent. As I illustrated you can recover something, since a lot of models use the same approach in calculating confidence intervals, but there is a lot of models too where some different approach is used.