How do I test if two (non-normal) distributions differ?

I have read about Student's t-test but it appears to work when we can assume that the original distributions are normally distributed. In my case, they are definitely not.

Also, if I have 13 distributions, do i need to do 13^2 tests?

-
@Glen_b The data is not discrete. Values range from -2 to 2. – Martin Velez Feb 26 '14 at 22:41

There are several senses in which "it depends".

(One potential concern is that it looks like the original data might perhaps be discrete; that should be clarified.)

1. depending on sample size, the non-normality may not be as big an issue as all that for the t-test.

2. If you're looking for any kind of differences in distribution, a two-sample goodness of fit test, such as the two-sample Kolmogorov-Smirnov test might be suitable (though other tests might be done instead).

3. If you're looking for location-type differences in a location-family, or scale differences in a scale family, or even just a P(X>Y)>P(Y>X) type relation, a Wilcoxon-Mann-Whitney two sample test might be suitable.

4. You might consider resampling tests such as permutation or bootstrap tests, if you can find a suitable statistic for the kind(s) of differences you want to have sensitivity to.

Also, if I have 13 distributions, do i need to do 13^2 tests?

Well, no.

Firstly, you don't need to test $A$ vs $B$ and $B$ vs $A$ (the second comparison is redundant).

Secondly, you don't need to test $A$ vs $A$.

Those two things cut the pairwise comparisons down from 169 to 78.

Thirdly, it would be much more usual (but not compulsory) to test collectively for any differences, and then, perhaps to look at pairwise differences in post-hoc pairwise tests if the first null was rejected.

For example, in place of a Wilcoxon-Mann-Whitney as in item 3. above, one might do a Kruskal-Wallis test, which is sensitive to any differences in location between groups.

There are also k-sample versions of the Kolmogorov-Smirnov test, and similar tests of some of the other two-sample goodness of fit tests might exist, or be constructed.

There are also k-sample versions of resampling tests, and of the t-test (i.e. ANOVA, which might be okay if the sample sizes are reasonably large).

It would be really nice to get more information about what we're dealing with and what kinds of differences you're most interested in; or failing that, to see Q-Q plots of some of the samples.

-
(+1) Just like to add that the WMW test bears an interpretation as a test for stochastic dominance if you're prepared to assume that the population CDFs don't cross. IMO people would more often want that if they knew about it. – Scortchi Feb 26 '14 at 13:30
@Scortchi - yes. It has yet other interpretations as well; most broadly it's a test of $P(X<Y) \neq \frac{1}{2}$. On the other hand, people should be aware of its non-transitivity property. – Glen_b Feb 26 '14 at 20:22
@Glen_b The data is not discrete. Values range from -2 to 2. – Martin Velez Feb 26 '14 at 22:18
Wow - important information! Are they bounded to that range (2.1 is impossible), or did it just happen that the values are in that range? – Glen_b Feb 26 '14 at 22:29
They are bounded to that range. – Martin Velez Feb 27 '14 at 0:22

Yes, I think you cannot do better than testing each distribution against the others...

If think that your question is related to this one :Comparison of 2 distributions

You advise you to use a Kolmogorov-Sminorv test or a Cramér-Von Mises test. They are both very classical adequation tests.

In R, function ks.testin stats package implements the first one. The second one can be found in packages like cramer.

To learn about these two tests : http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test http://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93von_Mises_criterion

-

You can try Kruskal–Wallis one-way analysis of variance

"It is used for comparing more than two samples that are independent, or not related"

Normality violations in ANOVA were discussed in
Rutherford Introducing Anova and Ancova: A GLM Approach 9.1.2 Normality violations

The first line there is "Although most sources report ANOVA ... as being robust with respect to violations of the normality assumption..."

-
Thanks! It appears that one should run this test before doing pairwise comparisons. – Martin Velez Feb 28 '14 at 20:58