Suppose I have two samples S1 and S2, but they are dependent samples. My aim is to compare if the mean of S1 is significantly smaller than the mean of S2. The samples are not normally distributed, so because of the distributional assumption, a non-parametric test (Wilcoxon sum-rank test, say) is preferred. However, these kind of non-parametric tests still require the samples to be independent.

My question is, is there a test that deals with dependent samples? How about sampled permutation test? For example, I have 5 data points in S1 and 10 data points in S2, and I can do the following (set counter = 0):

  1. Reshuffle the 15 points into 2 groups, each having 5 and 10 points
  2. Calculate the means of group 1 and group 2, and denote the difference as m1
  3. Compare with the observed mean difference m0: if m0 < m1, counter = counter + 1

Repeat the steps above for N=1000 times, and the p-value is counter/N.

Will this sampled permutation test handle dependent data well, compared with a single non-parametric test? Thanks!


The wilcoxon matched pairs test (http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test) may work here. Keep in mind that nonparametric tests rarely compare means; the wilcoxon matched pairs test, for example, is an hypothesis test concerning mean ranks of the underlying populations. It requires matched pairs, which is usually implied when we talk about "two sample dependent data." Your example has different sample sizes drawn from the two populations, so either your dependence structure is more complex than the typical paired case, or it was an oversight.

Your permutation test is appropriate for a one-tailed test of a mean difference between independent samples. It will be underpowered since it ignores the dependence structure. For example, consider an experiment in which 10 rats were run through a maze twice, with and without a drug. So the $i^{th}$ rat has two measurements, completion time with the drug, $t_D^{(i)}$, and time without, $t_C^{(i)}$. Each iteration of a permutation test in this case would randomly swap scores within each rat, not across rats as in your example. In other words, $t_C^{(i)}$ is swapped with $t_D^{(i)}$ with probability 0.5 for each $i$, but $t_C^{(i)}$ is never swapped with $t_D^{(j)}$. Simply shuffling all scores into two groups of ten would result in a test that suffers from reduced power relative to the more appropriate test.

  • $\begingroup$ thanks! yes, my situation for the two group of samples is quite complex: each data point is obtained by calculating from some vectors, and those vectors are shared which made the data points correlated in a complex way. I guess maybe there is no optimal established way to handle this situation? How to determine if one group is on average smaller than the other group? $\endgroup$ – alittleboy Jul 28 '13 at 2:15
  • $\begingroup$ Think carefully about your effect of interest, and then consider what "no effect" means -- simulate based on this scenario (if possible). Perhaps based on the underlying vectors you describe? In this situation permutation tests can be a tough sell to a non-statistical audience for two reasons (1) "The results change every time???" and (2) you won't be following a method verbatim from a standard reference, textbook, etc. Is there a more established statistical analysis for the data you have collected? Look to the literature, and perhaps supplement with a carefully designed permutation test. $\endgroup$ – ahfoss Jul 28 '13 at 2:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.