Assess calibration of a density forecast by Kolmogorov-Smirnov test on PIT of realized values

Question

According to Elliott & Timmermann "Economic Forecasting" (2016) p. 429-430,

Calibration requires that if a density forecast assigns a certain probability to an event, then the event should occur with the stated probability over successive observations.
<...>
For any <...> event, $A$, if the associated density forecast $\int_A p_Y (y|z)(y)\ dy = p$, calibration requires that $P(y_{t+1} \in A)$ is indeed equal to $p$, conditional on the same information.

I wonder how one could assess calibration of a density forecast. I think Kolmogorov-Smirnov test applied on the probability integral transform (PIT) of realized values vs. the theoretical Uniform[0,1] distribution could be used for that, following Section 3 of Diebold et al. (1998). The PIT would be based on the distribution that is implied by the density forecast. However, use of the test is not mentioned in the textbook (it says Most attempts to examine calibration lead to informal rather than formal hypothesis tests and goes on to discuss some difficulties with assessing calibration), so I am probably missing something.

Q: Does Kolmogorov-Smirnov test applied on the probability integral transform (PIT) of realized values vs. the theoretical Uniform[0,1] distribution assess calibration of a density forecast?

References

• Diebold, F. X., Gunther, T. A., & Tay, A. S. (1998). Evaluating Density Forecasts. International Economic Review, 39(4), 863-883.
• Graham, E., & Timmermann, A. (2016). Economic Forecasting. Princeton University Press.

Answer 1

First off, the PIT is not uniform in the discrete case, so the answer is "no" here. You can use randomization, an approach which apparently has been invented multiple times independently, but I suspect you are thinking of continuous densities, so let's assume this from now on.

The IMO canonical reference is Gneiting, Balabdaoui & Raftery (2007, Journal of the Royal Statistical Society: Series B). They define multiple different reasonable flavors of calibration (probabilistic, marginal and exceedance), show by examples that they are indeed logically independent and note that

Probabilistic calibration is essentially equivalent to the uniformity of the PIT values.

So in principle, yes, you could use the K-S test here. (Shameless piece of self-promotion here: an alternative would be data-driven tests for uniformity, which I used in Kolassa, 2016, IJF.)

However, Gneiting et al. write:

Uniformity is usually assessed in an exploratory sense, and one way of doing this is by plotting the empirical CDF of the PIT values and comparing it with the CDF of the uniform distribution. ... However, the use of formal tests is often hindered by complex dependence structures, particularly in cases in which the PIT values are spatially aggregated.

Gneiting et al. go on to discuss other diagnostic tools for probabilistic and other flavors of calibration, and give a number of pointers to literature. (Their main recommendation is proper scoring rules.) I would recommend you take a look at this paper, and at subsequent papers that cite it.