Testing if two non-normal distributions are significantly different (K-S or Wilcoxon or both?)

Question

The Big Question

I'm trying to decide between K-S and Wilcoxon (or something I haven't thought of yet) for two data sets I want to prove are significantly/insignificantly different from one another

"Significantly different in what way?" you ask? Well, I want to compare if they trend differently, so I guess that means I want to see if they are from the same population. I think comparing the means in this case might not be a valuable way of testing this, and my data is non-normal I ruled out Student's t (but that might just be my ignorance).

Because my sample size is smallish (see below in the How I've Treated My Data heading), and because the distribution is non-normal, I was thinking that either the Kolmogorov-Smirnov or the Wilcoxon signed-rank test would be best to compare the two distributions.

Are there other better tests that suit my needs better?

How I've Treated My Data

I'm not terribly sure how relevant this is, but it JUST MIGHT be of use to you folks, so here it goes:

So, I have two groups of paired histograms, that is to say that each column represents a range of values (e.g <1, 1-5,5-10...etc) and the value of each cell is the number of elements in my sample within that value range.

There are 9 pairs in group 1, 15 pairs in group 2

EDIT: Each pair represents a gene, and the first row of each pair is the wildtype data while the second row is the mutated data. Each group represents a family of genes.

[Group 1]
Pair 1-1,  11   0   12  0   58  1   72  8   18  31
Pair 1-2,  23   0   6   0   54  8   70  11  21  18
...
Pair n-1, ...
Pair n-2, ...

[Group 2]
Pair 1-1,  11   0   12  0   58  1   72  8   18  31
Pair 1-2,  23   0   6   0   54  8   70  11  21  18
...
Pair n-1, ...
Pair n-2, ...

I want to see the relative change of the histograms between each pair, so I've normalised them to one another using this formula (because I often have one of the value equalling zero while the other doesn't).

EDIT: I want to see the relative change between these histograms to see the relative increase or decrease between the two.

$$ \frac{( {A} - {B })} {\frac{A+B}{2}} $$

Which resulted in a table that looks something like

[Group 1]
Pair 1     10.00    0.32    0.71    -0.03   -0.07   0.15    -0.53   -0.67   1.56    0.00
...
Pair n     ....


[Group 2]
Pair 1     10.00    0.32    0.71    -0.03   -0.07   0.15    -0.53   -0.67   1.56    0.00
...
Pair n     ....

I then averaged the data in every column two see the average normalised number of members in each category of each group.

That results in something that looks like

[Group 1]
0.00    -0.15   0.28    1.15    0.25    -0.54   -0.08   -0.46   0.25    0.00

[Group 2]
0.00    0.33    0.05    0.26    -0.03   0.03    -0.79   0.16    -0.26   0.00

It is these two distributions, one from each group, that I wish to compare.

EDIT: I hope to see the difference in distribution between the family of genes.

---

Answer 1

Kolmogorov-Smirnov and similar goodness-of-fit tests work best for detecting a shift in the bulk of the distribution. As I understand, the ordering of the genes in your data set does not really matter. i.e. you would like to have equal power for detecting deviations regardless of the gene ordering.

Furthermore, your data is discrete and therefore the p-values returned by the Kolmogorov-Smirnov test would be inaccurate.

For testing goodness-of-fit between two discrete categorical distributions, the standard answer is to use the chi-squared test. However, the existence of genes with very few samples means that the chi-squared p-value would be inaccurate - the standard rule of thumb is that each bin needs to have at least ~5 samples.

Instead, I suggest using Fisher's Exact test for general n by 2 tables. This test uses simulations and provides a good estimate of the p-value. Here is an example of an R package implementing the general n by m test.

Answer 2

Since you clearly state that your main goal is to test whether both groups come from the same population, I do believe the K-S test would do fine. My guess is that Wilcoxon would be redundant if K-S test is not significant.

Answer 3

Both the Wilcoxon and the Kolmogorov Smirnov tests assume continuous distributions.

You seem to have ordered categorical data. As it stands, you can't use the 'ordinary' versions of these tests because their null distributions (i.e. what's in the tables) are based on that continuity assumption.

Both tests can be adapted to the ordered categorical (i.e. heavily tied) case, but the distributions depend on the pattern of ties. In small samples it might be possible to compute the permutation distribution, in medium samples you might sample the permutation distribution (randomization test), in large samples there is a normal approximation that incorporates the effect of ties for the Wilcoxon.

There are other tests that are possible, but a suitably adjusted version of whichever of those tests meet your requirements** should be okay.

** What you want to find out isn't particularly clear