0
$\begingroup$

I have a forecast which involves sampling a probability distribution and therefore each time I run the forecast there is some random variation between results. If I run the forecast many times, how do I compute the expected forecast, 5% and 95% confidence intervals using the ensemble of results?

Two options I have tried

(1) At each time step compute the 0.5, 0.025 and 0.975th quantiles across all forecasts.

(2) Take the sum over all time steps and use the forecasts where this sum corresponds to 0.5, 0.025, 0.975th quantile of all sums?

I am pretty sure both methods are incorrect.

The first because it involves choosing from each forecast at each time step. Each forecast is an independent realization and so it feels like I should be considering each forecast independently. In any case, the confidence intervals I get when I use this method are very wide, much wider than the max variation in the individual forecasts.

The second option also seems incorrect. When I use this method the confidence intervals may cross. Furthermore, who is to say the forecast I choose to represent the ith confidence interval will still represent the ith confidence interval when I run the ith+1 time step.

Hoping someone can explain the floors in my logic and help me figure out the correct procedure.

Improve this question
$\endgroup$
6
  • $\begingroup$ Not an answer to your broader question, but if you are aiming for 95% CIs, you probably need to be grabbing the 0.025 and 0.975 quantiles, not the 0.05 and 0.95 quantiles -- which would only contain 90% of the time. $\endgroup$
    – user5957401
    Jul 14, 2020 at 19:49
  • $\begingroup$ good point thanks! $\endgroup$
    – fen
    Jul 14, 2020 at 19:50
  • $\begingroup$ Given you are forecasting, it sounds like you probably want to compute prediction intervals rather than confidence intervals. Could you elaborate on what this interval is intended to do? $\endgroup$
    – whuber ♦
    Jul 14, 2020 at 19:52
  • 1
    $\begingroup$ @whuber I think you might be right, I am looking for a measure of uncertainty on my forecast. i.e. define the region about my forecast where the true value will fall 95% of the time. $\endgroup$
    – fen
    Jul 14, 2020 at 20:03
  • $\begingroup$ Do you expect the underlying probability to be constant so that all you have is sampling variance, or are you trying to identify changes in probability over time? $\endgroup$
    – EdM
    Jul 14, 2020 at 20:21

1 Answer 1

Reset to default
0
$\begingroup$

If I have understand the question correctly, this is a case of Ensemble Forecasting, and I believe the goal is to find Prediction rather than Confidence Intervals. Given a set of $n$ predictions $\hat{y}_{1, t+h}, \hat{y}_{2, t+h}, .., \hat{y}_{n, t+h}$ for $h$ timesteps ahead:

  • Expected forecast: average of the set of predictions
  • 50% Prediction Interval: median of the predictions
  • 95% Prediction Interval: Find the 2.5% and 97.5% quantiles of the inverse empirical cumulative distribution of $\hat{y}_{i, t+h}$, there is 95% chance that the prediction falls in this interval.
Improve this answer
$\endgroup$
3
  • $\begingroup$ This is great thank you and explains some of the confusion I have been having. After some analysis of my results, it seems sensible that the expected forecast should be the average since it is more reflective of the ensemble than the median. Can you put this on a more robust footing for me? Why mean gives Expected and median gives 50% PI. My floored intuition would be that for a unimodal symmetric distribution, one would want to report the "most likely" outcome, which I would naively say is the median. Looking at my results, however, this is clearly not the case. $\endgroup$
    – fen
    Jul 22, 2020 at 15:34
  • $\begingroup$ clarification on my previous comment, I meant asymmetric distribution $\endgroup$
    – fen
    Jul 22, 2020 at 15:47
  • $\begingroup$ By definition 'mean' corresponds to the 'expected' value (or average). Note than in this case, it's most probably the conditional mean given the available information until time $t$. Median is literally the midpoint of a frequency distribution; by default 50% of the population lies above/ below. If you have a Gaussian distribution these probably are the same; if your distribution is skewed, however, they'll differ. $\endgroup$
    – Akylas Stratigakos
    Jul 23, 2020 at 6:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.