How do I test for a "location shift" in something like the mean or median if the shape of the distribution has changed quite significantly between groups?

Often an experimental intervention seems to completely change the distribution of the data. For example, If I measure limb length in control and toxin-exposed animals, the control group sample might appear to be drawn from a symmetric, normal distribution but the experimental group might be heavy-tailed and have much more dispersion.

All the statistical tests I am familiar with either assume that the data is drawn from normally-distributed populations or that the data is drawn from two populations with non-normal but identical distribution shapes.

I can do something like a two-sample Kolmogorov-Smirnov test which tells me that my groups differ. But this isn't telling me anything about location, right?

  • $\begingroup$ @Glen_b You're thinking of the goodness-of-fit test; there's a K-S test for comparing two samples too. $\endgroup$ – Scortchi - Reinstate Monica Jun 15 '13 at 10:16
  • $\begingroup$ Indeed, there is, but it's almost invariably referred to as the two sample K-S test - to avoid exactly this problem (or sometimes, just 'the Smirnov test'), which is what threw me. Thanks. $\endgroup$ – Glen_b -Reinstate Monica Jun 15 '13 at 13:53
  • 1
    $\begingroup$ Yashka - The two sample K-S test can tell you about location-shift type difference; you can get either a point estimate or a confidence interval. It won't tell you the individual locations though. $\endgroup$ – Glen_b -Reinstate Monica Jun 15 '13 at 13:55

You are correct that a K-S test deals with the shape of the populations and not the location.

Before I suggest a strategy to try, consider this: if there is a substantial change in the shape of the distribution then is the location really what you want to know about? Is a test of a hypothesis relating to the location of any inferential utility?

If you answer yes to those questions, then if you have lots of data, current and past, try a couple of simple transformations to see if you can make the likely population shapes similar. (I always like a log transformation for heteroscedastic data.) Once the populations are tolerably similar use either a permutations test or a Student's t-test on the location and trust that the robustness of those tests will deal with any non-identicalness and non-normality respectively.

Your conclusions should be carefully calibrated taking the nature of the data in mind. A significance test approach where the P-value is interpreted as an index of the strength of evidence against the null is much more useful in such circumstances than a Neyman-Pearson hypothesis test which yields a binary outcome.

  • $\begingroup$ biologically, it makes sense that the bone has an average size under normal conditions. The fact that our toxin makes it shorter with more variability than exists without an intervention is not really surprising. If this were a factor that made the bone bigger with increased variability I would simply use a log transform. $\endgroup$ – Yashka Oreza Jun 15 '13 at 0:20
  • $\begingroup$ Good point. Perhaps a reciprocal transform would help, but I have no experience with it. $\endgroup$ – Michael Lew - reinstate Monica Jun 15 '13 at 1:50
  • $\begingroup$ I would commend quantile-quantile plotting for initial comparison of distributions here. The original key paper Wilk, M. B. and Gnanadesikan, R.. 1968. Probability plotting methods for the analysis of data. Biometrika 55: 1-17 remains a source of inspiration. stata-journal.com/sjpdf.html?articlenum=gr0027 gives some more recent references. $\endgroup$ – Nick Cox Jun 15 '13 at 11:32

The modified Mathisen test (Hettmansperger & McKean (1998), Robust Nonparametric Statistical Methods, Ch. 2.11) tests for equal medians under the null hypothesis while making no assumptions about the distributions of either sample other than that they're continuous.

But do bear in mind what @Michael said—a difference in medians is usually interesting only when you think there's a location parameter shifting. More often in your situation people use the Mann–Whitney–Wilcoxon test to assess stochastic dominance: that's whether one group has a lower or higher cumulative distribution function of the response for all its values & would correspond to the idea of seeing whether the toxin has a good, a bad, or not much effect. The assumption here is that the population cdfs don't cross, & is usually checked by examining a plot of the sample cdfs.

  • $\begingroup$ I had not heard of stochastic dominance before but it seems like a good concept to apply to these situations. $\endgroup$ – Yashka Oreza Jun 17 '13 at 6:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.