What are some ways to see how different two discrete probability distributions are? I have two samples that can only take value from 1-4, in increments of 0.5, and I'm trying to compare their distributions without using Kolmogorov-Smirnov test, as that is primarily designed for continuous distributions. I am also aware of the Jensen Shannon divergence. Are there any other methods?
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ Are there particular kinds of alternatives you're most interested in being able to pick up (and so would like to have more power to detect), or is any difference in distribution pretty much of equal concern? $\endgroup$– Glen_bCommented Nov 24, 2022 at 22:48
-
1$\begingroup$ To give more context, the distributions I'm comparing are distributions of resampled modes (so bootstrapping with the modes). So if one distribution has more probability mass around a lower mode value versus a higher one, that would be nice to detect. $\endgroup$– user373876Commented Nov 24, 2022 at 22:49
-
$\begingroup$ Chi-squared is the standard technique. $\endgroup$– whuber ♦Commented Nov 25, 2022 at 16:04
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
For general differences in distribution there's the chi-squared test. (Alternatively, you could adapt the Kolmogorov-Smirnov to the discrete case via a permutation test, which should reduce the power loss from substantial overconservatism - unless, perhaps, a lot of the distribution is focused into only a few outcomes.)
If there's a preference to detect tendency to be lower or higher you would use a test that focuses its power on detecting those sorts of alternatives. There's a number of possibilities, including a Wilcoxon-Mann-Whitney. As with the Kolmogorov-Smirnov, the heavy ties need to be dealt with. In larger samples you might consider using the normal approximation with the variance adjustment for ties, but more generally you could use a permutation test. Indeed any of the mentioned tests could be implemented as a permutation test.
Somewhat in between the chi-squared and the other options would be a Neyman-Barton-style smooth test of goodness of fit, focusing on the low-order components; a focus on only the lowest component is very similar to a z-test of means.
