How to evaluate the accuracy of a probability distribution? I've trained a Gaussian Bayesian Network. If I feed input values for the parent variables of my output variable, I get a normal distribution. How can I quantify the accuracy of this distribution when confronted to the real data? Should I just use the variance?
 A: There are different ways of assessing the accuracy of predictive densities.
The simplest would be a Probability Integral Transform: evaluate your predicted CDF at the observed outcome. If your predicted CDF is the correct density, then the result will be uniformly distributed on [0,1]. (This is just a different way of looking at the definition of the CDF.) This is rather simple to understand, but since your CDF being correct is a sufficient condition for the PIT to be uniform, not a necessary one, your predicted CDF may be wrong even if your PIT is uniform (Gneiting et al., 2007).
A more correct, but less easily understood, approach is to use proper scoring-rules. These are functions of your predicted density and the actual outcome that will be minimized in expectation by the correct density. See Gneiting et al. (2007) and Gneiting & Katzfuss (2014). (Note that there are different conventions in the literature: Gneiting et al. use "negatively oriented" scoring rules, where smaller is better, but other people use "positively oriented" ones.)
