# Forecast confidence intervals from multiple realizations

I have a forecast which involves sampling a probability distribution and therefore each time I run the forecast there is some random variation between results. If I run the forecast many times, how do I compute the expected forecast, 5% and 95% confidence intervals using the ensemble of results?

Two options I have tried

(1) At each time step compute the 0.5, 0.025 and 0.975th quantiles across all forecasts.

(2) Take the sum over all time steps and use the forecasts where this sum corresponds to 0.5, 0.025, 0.975th quantile of all sums?

I am pretty sure both methods are incorrect.

The first because it involves choosing from each forecast at each time step. Each forecast is an independent realization and so it feels like I should be considering each forecast independently. In any case, the confidence intervals I get when I use this method are very wide, much wider than the max variation in the individual forecasts.

The second option also seems incorrect. When I use this method the confidence intervals may cross. Furthermore, who is to say the forecast I choose to represent the ith confidence interval will still represent the ith confidence interval when I run the ith+1 time step.

Hoping someone can explain the floors in my logic and help me figure out the correct procedure.

• Not an answer to your broader question, but if you are aiming for 95% CIs, you probably need to be grabbing the 0.025 and 0.975 quantiles, not the 0.05 and 0.95 quantiles -- which would only contain 90% of the time. Jul 14, 2020 at 19:49
• good point thanks!
– fen
Jul 14, 2020 at 19:50
• Given you are forecasting, it sounds like you probably want to compute prediction intervals rather than confidence intervals. Could you elaborate on what this interval is intended to do?
– whuber
Jul 14, 2020 at 19:52
• @whuber I think you might be right, I am looking for a measure of uncertainty on my forecast. i.e. define the region about my forecast where the true value will fall 95% of the time.
– fen
Jul 14, 2020 at 20:03
• Do you expect the underlying probability to be constant so that all you have is sampling variance, or are you trying to identify changes in probability over time?
– EdM
Jul 14, 2020 at 20:21

If I have understand the question correctly, this is a case of Ensemble Forecasting, and I believe the goal is to find Prediction rather than Confidence Intervals. Given a set of $$n$$ predictions $$\hat{y}_{1, t+h}, \hat{y}_{2, t+h}, .., \hat{y}_{n, t+h}$$ for $$h$$ timesteps ahead:
• 95% Prediction Interval: Find the 2.5% and 97.5% quantiles of the inverse empirical cumulative distribution of $$\hat{y}_{i, t+h}$$, there is 95% chance that the prediction falls in this interval.
• By definition 'mean' corresponds to the 'expected' value (or average). Note than in this case, it's most probably the conditional mean given the available information until time $t$. Median is literally the midpoint of a frequency distribution; by default 50% of the population lies above/ below. If you have a Gaussian distribution these probably are the same; if your distribution is skewed, however, they'll differ. Jul 23, 2020 at 6:47