I've come across some recent demand forecasting approaches that present methods where instead of generating just a point forecast, the model outputs a set of forecast quantiles, or a distribution of counts.
Having trained such a model - how do you evaluate its performance?
Once your actuals start coming in, you only have one value per time step which you could compare to the mean or to the median of the output, but against what do you compare the 'rest' of your output? How do you evaluate the other quantiles besides your 50% quantile? Or how do you evaluate the other parameters of your distribution other than the mean?