# Finding a Metric for Resampling a Distribution

I have a distribution of a large group. It's uniform, and looks like this:

I have experimental results, where I look to observe members of this group. I don't in any way expect a majority of them to be present.

My experimental results have a sampling distribution that looks like this:

To me, this looks like a nonrandom sample from the main group. And I'd like to show this. I want to resample random selection from the main group, but my problem is I don't know what kind of metric to assess.

Mean/standard deviation/variance all look rather similar. Is there a better metric I could use to assess what my random resampled groups look like, and show that the distribution of the sampled group is nothing like my resampled groups?

Or is there a test I could use in this case?

• Kolmogorov-Smirnov tests are used for comparing distributions. Oct 5, 2020 at 16:08
• One difference between the two histograms you show is skewness. The first is neither uniform nor symmetrical; the second seems symmetrical. Have you tried looking at a measure of skewness as metric for a permutation test? Oct 5, 2020 at 18:01
• The skew was 0.081 and 0.053, it seems different enough. I've heard of the kolmogorov-smirnov test before. I'll try that as a comparator. Oct 5, 2020 at 18:49
• Another possibility is to compare their entropies. Check this question stats.stackexchange.com/questions/28178. NRH's answer explains how to compute the entropy of an empirical distribution. Oct 11, 2020 at 18:24
• Ordinarily, it would be a little dicey to propose the metric after noting there might be a difference: you would have to work a little to show this doesn't skew the conclusion. But in this case, with such a large sample where it is blatantly obvious the distribution is different, you don't need a test at all. For those who insist on testing, a simple test, making few assumptions, whose results are not strongly dependent on these issues, is the chi-squared. Four bins with cutpoints at $1/4,2/4,3/4$ would work fine.
– whuber
Sep 20, 2021 at 22:00

After repeating $$R$$ times, draw the histogram based on the $$R$$ KS values, and overplot ( a red point) the KS value based on your two histograms.
You can calculate a simulated p-value from this! One often uses a value such as $$R=9999$$, including the KS value from your data, this gives a total sample for calculation pf p-value $$R+1=10000$$. The p-value is the fraction of values larger or equal to the one from the experiment (the red point).