I have a forecast which involves sampling a probability distribution and therefore each time I run the forecast there is some random variation between results. If I run the forecast many times, how do I compute the expected forecast, 5% and 95% confidence intervals using the ensemble of results?
Two options I have tried
(1) At each time step compute the 0.5, 0.025 and 0.975th quantiles across all forecasts.
(2) Take the sum over all time steps and use the forecasts where this sum corresponds to 0.5, 0.025, 0.975th quantile of all sums?
I am pretty sure both methods are incorrect.
The first because it involves choosing from each forecast at each time step. Each forecast is an independent realization and so it feels like I should be considering each forecast independently. In any case, the confidence intervals I get when I use this method are very wide, much wider than the max variation in the individual forecasts.
The second option also seems incorrect. When I use this method the confidence intervals may cross. Furthermore, who is to say the forecast I choose to represent the ith confidence interval will still represent the ith confidence interval when I run the ith+1 time step.
Hoping someone can explain the floors in my logic and help me figure out the correct procedure.