# Checking for statistical "agreement" between multiple arbitrary distributions

I have $$N$$ methods that each report a posterior probability density for the measurement of $$X$$.

When $$N=2$$ and the posteriors are both approximately Gaussian, one can check whether the posteriors agree by comparing their means divided by their standard deviations. This can reveal for example whether the two measurements are more than, say, $$5\sigma$$ apart and hence significantly different.

What can be analogously done when $$N\geq 2$$ and/or any of the distributions are not approximately Gaussian?

If the general case is difficult, in the special case where one of the distributions is non-normally distributed but the rest are normally distributed, is there something that can be done there?

• Could you give some more context, show us a plot? Assuming your posteriors are one-dim, you could sample from them and use some form of anova ... Mar 8, 2021 at 9:34

Another possible (non-symmetric) way of measuring the "distance" between any 2 distributions is the KL-Divergence.

In this context, this MathOverflow post discusses specifically how to generalize the KL for more than 2 distributions. User Carlo Beenakker provides some references worth checking! For example (all credit goes to him):

Also intuitively, I'd say that building up a sort of "Confusion matrix" $$M$$ with the KL divergences where $$M_{a, b} = D_{KL}(X_a \| X_b )$$ could be helpful (if you don't need to reduce the analysis to a single scalar).