I am comparing a feature between correct & incorrect trials. There are N=1194 paired samples $\{x_i,y_i| i=1,2,...N\}$, where x is for correct trials and y is incorrect trials.

I am using a nonparametric Mann-Whitney test to test the hypothesis $x>y$. (although I get similar results with a normal t-test)

My results show that x>y with P-value of P=2.3e-11 (very confident). Furthermore, by using a log-transform & getting the difference of the medians, I get that median(x) is 48% greater than median(y)

So far, these are great results. But then I plot my data, as shown below. Left figure is a boxplot with grey lines being individual samples. Middle plot are histograms of x,y (top) and their paired differences (bottom). Right plot is a scatterplot of the data {x,y}.


Although looking at the bottom histogram of paired differences, x is clearly greater than y, looking at the boxplot & scatterplot it seems like theres a huge amount of variability. Thus my P-value of P=2.3e-11 seems EXTREMELY confident, and the ratio of medians of 48% seems too high.

Am I doing this right?

  • $\begingroup$ Some reasons to explain this are: 1) MW whitney tests & t-tests only describe the difference of the mean; they say nothing about the variablity about the mean. 2) I have a huge amount of data (N=1194) to give me very high power) 3) My data is positive & negative, so the 'ratio' of medians is meaningless. $\endgroup$ – DankMasterDan Nov 28 '14 at 21:17
  • $\begingroup$ (2) and (3) are correct, but (1) definitely is not: all statistical tests--and these two are archetypical examples--specifically account for variability. However, the variability they address is the sampling variability of the statistic, not the variability of the data themselves. See our threads about sampling distributions. $\endgroup$ – whuber Nov 28 '14 at 22:32
  • $\begingroup$ @whuber do you think its reasonable to get such a low P value for a scatterplot/boxplot that visually looks so equal? $\endgroup$ – DankMasterDan Nov 28 '14 at 23:09
  • 1
    $\begingroup$ Yes: the histogram displays an SD of about 0.5, whence the standard error of the mean will be approximately $0.5/\sqrt{1100}\approx 0.015$. A mean difference of $0.077$ would therefore be about $5$ se's away from $0$, which is highly significant. $\endgroup$ – whuber Nov 29 '14 at 3:21
  • 1
    $\begingroup$ Effect size. For the M-W, this is best measured as an estimate of the chance that a random $X$ will exceed a random $Y$. $\endgroup$ – whuber Dec 1 '14 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.