3
$\begingroup$

I have a distribution of a large group. It's uniform, and looks like this:

enter image description here

I have experimental results, where I look to observe members of this group. I don't in any way expect a majority of them to be present.

My experimental results have a sampling distribution that looks like this:

enter image description here

To me, this looks like a nonrandom sample from the main group. And I'd like to show this. I want to resample random selection from the main group, but my problem is I don't know what kind of metric to assess.

Mean/standard deviation/variance all look rather similar. Is there a better metric I could use to assess what my random resampled groups look like, and show that the distribution of the sampled group is nothing like my resampled groups?

Or is there a test I could use in this case?

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ Kolmogorov-Smirnov tests are used for comparing distributions. $\endgroup$
    – Xi'an
    Commented Oct 5, 2020 at 16:08
  • $\begingroup$ One difference between the two histograms you show is skewness. The first is neither uniform nor symmetrical; the second seems symmetrical. Have you tried looking at a measure of skewness as metric for a permutation test? $\endgroup$
    – BruceET
    Commented Oct 5, 2020 at 18:01
  • $\begingroup$ The skew was 0.081 and 0.053, it seems different enough. I've heard of the kolmogorov-smirnov test before. I'll try that as a comparator. $\endgroup$
    – Estif
    Commented Oct 5, 2020 at 18:49
  • $\begingroup$ Another possibility is to compare their entropies. Check this question stats.stackexchange.com/questions/28178. NRH's answer explains how to compute the entropy of an empirical distribution. $\endgroup$
    – jpneto
    Commented Oct 11, 2020 at 18:24
  • $\begingroup$ Ordinarily, it would be a little dicey to propose the metric after noting there might be a difference: you would have to work a little to show this doesn't skew the conclusion. But in this case, with such a large sample where it is blatantly obvious the distribution is different, you don't need a test at all. For those who insist on testing, a simple test, making few assumptions, whose results are not strongly dependent on these issues, is the chi-squared. Four bins with cutpoints at $1/4,2/4,3/4$ would work fine. $\endgroup$
    – whuber ♦
    Commented Sep 20, 2021 at 22:00

1 Answer 1

Reset to default
2
$\begingroup$

There is little information in your question ... a good metric would be one you cares about, which is important for your use of this. You did not tell us ... But you seem to have large samples (as judged by the y-axis on your histograms). So for a general test, you can base it on the KS (Kolmogorov-Smirnov) statistic.

Neither did you tell if you used smpling with or without replacement, but assuming the last: You can find the null distribution of the KS statistic, for your data, by simulation. Repeat:

  • Draw (without replacement) a sample size N (= sample size for your experimental results)

  • Calculate the KS statistic for comparing distribution of your group (from first histogram) to distribution of simulated sample

After repeating $R$ times, draw the histogram based on the $R$ KS values, and overplot ( a red point) the KS value based on your two histograms.

You can calculate a simulated p-value from this! One often uses a value such as $R=9999$, including the KS value from your data, this gives a total sample for calculation pf p-value $R+1=10000$. The p-value is the fraction of values larger or equal to the one from the experiment (the red point).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.