Empirically validating a forecast distribution

Question

I have a family of models that give me a forecast distribution for the next observation in a time series. So given observations $O_1, \dots, O_T$, I can calibrate the model and get a distribution for $O_{T+1}$. My observations are continuous random variables. How can I check which model provides the best forecast for the historic data? At every time point I have a distribution (which will be different for ever time $T$) and only a single observation.

I was thinking to calculate a few quantiles at every time $T$ (the quantiles should be the same, their value will be different obviously), which defines "buckets" for the value of the random variable. And then I would count the realizations falling into all these buckets, expecting to get proportional numbers for the optimal forecasts. Does this approach work? And how do I optimally choose the quantiles?

I would appreciate references in the answers, because I need to use this in an academic paper.

Answer 1

What you are doing is referred to as density forecasting, as you are not only forecasting single values (point forecasts) or intervals, but whole distributions or densities.

Even evaluating point forecasts is a non-trivial question, and there are long discussions about MAPE, wMAPE, RMSE, MASE or similar accuracy measurements as well as about tests for better predictive power or Forecast Encompassing tests - all in the seemingly simple context of assessing which one of two methods yields better point forecasts. So things get even more complicated for interval or density forecasts. While there are specific publications about point forecast evaluation, I am not aware of comparable work on evaluating density forecasts.

I would suggest that you read through recent papers on density forecasts and see how the authors evaluated their density forecasts. (Depending on what exactly you are doing, you may find even more inspiration and/or possible reviewers among the authors.) I would start by searching for "density forecasting" or similar keywords in the International Journal of Forecasting. Good luck!