Effect size, as estimated by Cohen's $\delta$, is:

$d=\frac{\hat x_1 - \hat x_2}{\hat s_{pooled}}$

Since the numerator is an estimate of difference in location, and the denominator is an estimate of the spread, I have always thought it reasonable to replace the top with difference in medians and the bottom with interquartile range:

$m=\frac{Median(X_1) - Median(X_2)}{IQR(X_1\ and\ X_2)}$

But I cannot find any references to justify this. Does this sound reasonable?

The type of application I'm looking for here is the often encounter situation where I'm doing test of location with small data ($n<20$). I compute the t-statistic and generate its null distribution with permutation. This gives me nice p-values. But for numerical estimation of effect magnitude, I thought this $m$ quantity may be helpful in describing the size of the effect magnitude.

Edit: Added clarification

  • 1
    $\begingroup$ I haven't heard of this either but if you're in a field of study where medians and IQR are nearly universally used there might be little wrong with this approach. Some simulations could easily demonstrate so. But don't call it anything like d. Given that the question is about the reasonableness of the approach you need to add why you do it. $\endgroup$ – John Nov 2 '14 at 20:02
  • $\begingroup$ How about Cohen's U3? $\endgroup$ – jona Nov 2 '14 at 20:35
  • $\begingroup$ For what it's worth, sample medians are approximately normally distributed: web.williams.edu/Mathematics/sjmiller/public_html/BrownClasses/…. That reference also derives the density of an arbitrary order statistic, in turn referring to John E. Freund’s Mathematical Statistics with Applications, seventh edition, so maybe that book also gives the distribution of a difference in order statistics. Then if it's something "nice" you can derive an analytical distribution for what you proposed $\endgroup$ – shadowtalker Nov 3 '14 at 10:09
  • 1
    $\begingroup$ And this (gated) paper seems to offer a general approach to testing differences in quantiles from different samples link.springer.com/article/10.1007%2Fs00362-007-0058-3 $\endgroup$ – shadowtalker Nov 3 '14 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.