The first thing that comes to mind when comparing some property of two samples is probably the independent samples t-test. Sometimes, people point out that this has the inherent assumption of the t-distribution. Does any one know of a non-parametric alternative to the independent samples t-test?
R. A. Fisher justified using the t-test because it was a reasonable approximation to the permutation test under certain conditions and the t-test was simpler to compute. With modern computers the computation of the permutation test (or the estimation by sampling from the permutation distribution) is no longer something to be avoided. The permutation test does not depend on the assumption of normality and is not restricted to comparing means (it can be used to compare means, but it can be used with medians, ranges, or other functions of the data that may be of interest).
So one non-parametric alternative is the permutation test (some of the other common non-parametric methods are actually special cases of a permutation test).
For small samples you can calculate every possible computation, but it is more common to use resampling techniques to estimate the permutation distribution. The basic steps:
- Choose the statistic of interest (difference of means, difference of medians, etc.) and compute it for the original data.
- Merge all the data together then randomly divide it into 2 groups of the same sizes as the original. Compute the desired test statistic on this random grouping.
- Repeat step 2 a bunch of times (until the total number of stats is around 1,000 or 10,000).
- Compare the original statistic to the distribution of the permuted statistics (the p-value is the proportion of test statistics that are as extreme or more extreme than the original).