# Deriving risk estimates using forecasting confidence limits and out of sample hold-out cases

I was hoping for some advice. I use SAS for automatic forecasting (I have a large number of forecasts to complete in a limited timeframe).

As part of the forecast output from SAS, I get a mid-point (median or mean), and an upper and lower confidence limit for each forecast. This is determined at a pre-specified level (i.e. 95%). Evidently, the confidence limits are derived statistically. I appreciate that within the upper and lower limits there is a range of potential forecasting results that could occur.

Based on my interpretation, values close to the upper and lower limits of the forecasts will be less likely because they depart a fair way from the mean/median and are close to the tails in terms of the distribution of possible forecasted values.

In a number of potential forecasting scenarios I face each month, I have a target the business needs to achieve by the end of the next financial year (june 30, 2012). I also have a forecast of the likely future value + upper and lower forecast limits for end of year (june 30, 2012) derived using moderate to long term historical performance. I need to quantify the probability of reaching a target given a forecasted result, using the point where the target is located relative to the mean/median and also the upper and lower forecast limits. For example, if the series has a mean forecast of 50, a LCL of 0 and UCL of 100 and the target is 75, I need to quantify the probability this target will be met at June 30, 2012.

It seems to me I can use the fact the target of 75 falls on the 75th percentile relative to the forecast upper and lower forecast limits.

• Is it fair to say that there is a 25% chance of hitting the target under an assumed uniform distribution?
• Or is it more appropriate to assume the forecast series within the upper and lower limits is normally distributed with a greater concentration of values in the CDF closer to the mean?
• Or is this model dependent (i.e. different for arima and esm models)

Also, as a rule of thumb how many holdouts do people use for

1. a series with 12 months (I am using 10% or 1)
2. a series with 24 months (I am using 10% or 2)
• What do you use the holdouts for? How is that related to the previous question? – mpiktas Aug 3 '11 at 7:06
• May we assume that you meant ECM, i.e. error corection model, not esm? – mpiktas Aug 3 '11 at 7:07

In theory the confidence intervals are derived from the estimated distribution of the forecast. This means that the model gives estimate $\hat{y}$ and its estimated cdf $\hat{F}$. The 95%-confidence interval is then calculated as

$$(\hat{F}^{-1}(0.025),\hat{F}^{-1}(0.975))$$

If $\hat{F}$ would be available you could assign (with certain assumptions) probability for the target $y$ by simply reporting $1-\hat{F}(y)$ which will be the probability $P(\hat{y}>y)$, i.e. the probability that the forecast will be larger than the target.

However you do not have $\hat{F}$, you just have 2 values: $\hat{F}^{-1}(0.025)$ and $\hat{F}^{-1}(0.975)$. In general this is too little information to recover $\hat{F}$. On the other hand it helps to know that for a lot of models $\hat{F}$ is normal distribution. Usually:

$$\hat{y}\sim N(\mu_{\hat y},\sigma^2)$$

where $\mu_{\hat y}$ is the reported mean or median forecast. Since in this case the 95% confidence interval is

$$(\mu_{\hat y}-1.96\sigma,\mu_{\hat y}+1.96\sigma)$$

It is not hard then to recover $\sigma$ and for the target $y$ we have

$$P(\hat{y}>y)=1-\Phi\left(\frac{y-\mu_{\hat y}}{\sigma}\right)$$

where $\Phi$ is standard normal distribution.

This should be used with caution, since although a lot of models do use normal distribution for $\hat{F}$, this is not always the case. Furthermore it might be that for example $\log y$ was modelled, but confidence interval was reported for $y$.