Checking for statistical "agreement" between multiple arbitrary distributions

Question

I have $N$ methods that each report a posterior probability density for the measurement of $X$.

When $N=2$ and the posteriors are both approximately Gaussian, one can check whether the posteriors agree by comparing their means divided by their standard deviations. This can reveal for example whether the two measurements are more than, say, $5\sigma$ apart and hence significantly different.

What can be analogously done when $N\geq 2$ and/or any of the distributions are not approximately Gaussian?

If the general case is difficult, in the special case where one of the distributions is non-normally distributed but the rest are normally distributed, is there something that can be done there?

Answer 1

Another possible (non-symmetric) way of measuring the "distance" between any 2 distributions is the KL-Divergence.

In this context, this MathOverflow post discusses specifically how to generalize the KL for more than 2 distributions. User Carlo Beenakker provides some references worth checking! For example (all credit goes to him):

1. Information radius
2. Average divergence
3. Dissimilarity

---

Also intuitively, I'd say that building up a sort of "Confusion matrix" $M$ with the KL divergences where $M_{a, b} = D_{KL}(X_a \| X_b )$ could be helpful (if you don't need to reduce the analysis to a single scalar).

Another related metric worth checking is the Jensen-Shannon divergence.

Hope this helped a bit!