How to you measure the accuracy of a model that gives quantile forecasts or distributions of forecasts? I've come across some recent demand forecasting approaches that present methods where instead of generating just a point forecast, the model outputs a set of forecast quantiles, or a distribution of counts. 
Having trained such a model - how do you evaluate its performance?
Once your actuals start coming in, you only have one value per time step which you could compare to the mean or to the median of the output, but against what do you compare the 'rest' of your output? How do you evaluate the other quantiles besides your 50% quantile? Or how do you evaluate the other parameters of your distribution other than the mean? 
 A: The standard approach is to use probability scoring. See Gneiting and Katzfuss (2014) for some of the mathematical background.
One example of a probability scoring measure is quantile scoring based on the pinball loss function. For each time period throughout the forecast horizon, you compute the $0.01, 0.02, \dots, 0.99$ quantiles --- call these $q_1,\dots,q_{99}$, with $q_0=-\infty$ or the natural lower bound, and $q_{100}=\infty$ or the natural upper bound. These 99 values then define (approximately) the forecast densities.
For a quantile forecast $q_a$ with $a/100$ as the target quantile, the pinball loss  $L$ is defined as:
$$
L(q_a, y) = \begin{cases}
(1 - a/100) (q_a - y), & \text{if $y< q_a$};\\
a/100 (y - q_a), & \text{if $y\ge q_a$};
\end{cases}
$$
where $y$ is the observation used for verification, and $a = 1, 2, \dots, 99$.
Note that $L(q_{50},y)$ is equal to $0.5 |q_{50}-y|$, half the value of the absolute error. For other quantiles, the loss is not symmetric.
To evaluate the full predictive densities, this score is then averaged over all target quantiles, from 0.01 to 0.99, for all time periods over all forecast horizons. The lower the score, the better the forecasts are.
