1
$\begingroup$

In Matlab I did run a Wilkoxon rank sum test to compare two variables. The number of observation -for each variable- was very small (N=7; for this reason I did not run a t-test, as I could not reasonably assume normality of data). I don't know how to evaluate the result of Wilkoxon test: response is significant but I'm worried about the very small sample size (only 7 observation...did I run a meaningful test?). For this reason I computed the statistical power of the test which, as far as I know, is the correct way to evaluate the "goodness" of test response (Am I right?). I used the Matlab function sampsizepwr which wants as argument the type-of-test (for example z-test, t-test, Chi-square, etc...). I guess it is 'z' but I'm not sure (Matlab page on Wilcoxon seems to suggests that ranksum uses z-statistics).

Question 1: is it meaningful to compute the statistical power of Wilcoxon test, when I have very small samples size?

Question 2: is it correct to use z-statistics (as argument of function sampsizepwr) for computing power of Wilcoxon test?

$\endgroup$
  • $\begingroup$ Its not clear what the question is $\endgroup$ – Kozolovska Jan 25 '18 at 18:09
  • $\begingroup$ I apologise and try to make it clearer: is it correct to use 'z-statistic' as argument of function 'sampsizepwr' when testing the power of Wilkoxon test? $\endgroup$ – valerio_sperati Jan 25 '18 at 18:15
  • 1
    $\begingroup$ Let me guess: the result was not significant... a power analysis is rarely useful after peeking at the results. $\endgroup$ – Michael M Jan 25 '18 at 18:22
  • $\begingroup$ @MichaelM on the contrary! The Wilcoxon test was significant and power was beyond 95%!!! This very high power value was suspicious to me, because of the small sample size (N=7). But maybe only because I'm not used to compute this measure... $\endgroup$ – valerio_sperati Jan 26 '18 at 11:06
1
$\begingroup$

Sample size and/or power calculations are primarily a tool for planning an experiment or study. Post-hoc, i.e. after performing the test on the real data, the power was high enough (if the result was significant) or not high enough (if the result was insignificant), so there are only limited situations where this would make sense.

Nevertheless, to answer your question: Usually, one would pick the t-test to find the power of Wilcoxon's rank sum test and reduce this power by 5% or 10% (assuming normality). As @Glen_b has mentioned, simulation would be an alternative in order to avoid to make distributional assumtions (like uniform or normal). In practice however, power and sample size calculations are very approximate. This distributional assumtion usually only has a very small impact compared to other assumptions (e.g. on minimally relevant effect size).

$\endgroup$
  • 3
    $\begingroup$ Re the last sentence -- if people could assume normality, ... they'd probably be doing the t-test. So about the only thing you could guarantee is that this calculation won't give you the power for the rank sum test. Why would you not just work with the rank sum test? If the alternative can be specified enough to notionally work out the power, it's also specified enough to simulate from, so you don't even need to be able to compute the power algebraically. $\endgroup$ – Glen_b Jan 26 '18 at 14:20
  • $\begingroup$ @Glen_b: Agreed about the simulation part. I have added this option. $\endgroup$ – Michael M Jan 26 '18 at 16:26
  • $\begingroup$ Well, the issue is, if you're going to specify an effect (such as a location shift) at which you compute the power, simulation doesn't avoid distributional assumptions (the power very much depends on the distribution!). You still make distributional assumptions, what it saves you from is both needing to be able to compute it algebraically, and having to assume normality. e.g. we could compute the power at n1=20,n2=15, data having a t(5) distribution with scale parameter $σ$ and, and a location shift of $½σ$. Or you could assume exponential samples and the second sample with twice the scale. $\endgroup$ – Glen_b Jan 27 '18 at 4:09
  • $\begingroup$ Even though sample size/power calculations are ideally done up-front to inform how big of a dataset you need, it can be useful post-hoc. If you see no significant result in your experiment, it's very useful to know if the sample size was too small to find the effect to begin with (in which case your lack of a significant result has little meaning), or if your sample size was indeed large enough (in which case the lack of a significant result is evidence that there is no effect). In the former, more data might change your conclusions, in the latter, it will not. $\endgroup$ – Nuclear Wang Nov 20 '18 at 21:07
  • $\begingroup$ This will lead to a very complex multiple testing issue (or a bias, of course). $\endgroup$ – Michael M Nov 20 '18 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.