1
$\begingroup$

I have $N$ methods that each report a posterior probability density for the measurement of $X$.

When $N=2$ and the posteriors are both approximately Gaussian, one can check whether the posteriors agree by comparing their means divided by their standard deviations. This can reveal for example whether the two measurements are more than, say, $5\sigma$ apart and hence significantly different.

What can be analogously done when $N\geq 2$ and/or any of the distributions are not approximately Gaussian?

If the general case is difficult, in the special case where one of the distributions is non-normally distributed but the rest are normally distributed, is there something that can be done there?

$\endgroup$
1
  • $\begingroup$ Could you give some more context, show us a plot? Assuming your posteriors are one-dim, you could sample from them and use some form of anova ... $\endgroup$ Commented Mar 8, 2021 at 9:34

1 Answer 1

0
$\begingroup$

Another possible (non-symmetric) way of measuring the "distance" between any 2 distributions is the KL-Divergence.

In this context, this MathOverflow post discusses specifically how to generalize the KL for more than 2 distributions. User Carlo Beenakker provides some references worth checking! For example (all credit goes to him):

  1. Information radius
  2. Average divergence
  3. Dissimilarity

Also intuitively, I'd say that building up a sort of "Confusion matrix" $M$ with the KL divergences where $M_{a, b} = D_{KL}(X_a \| X_b )$ could be helpful (if you don't need to reduce the analysis to a single scalar).

Another related metric worth checking is the Jensen-Shannon divergence.

Hope this helped a bit!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.