Effect size to Wilcoxon signed rank test?

Question

Some authors (e.g. Pallant, 2007, p. 225; see image below) suggest to calculate the effect size for a Wilcoxon signed rank test by dividing the test statistic by the square root of the number of observations:

$r = \frac{Z}{\sqrt{n_x + n_y}}$

Z is the test statistic output by SPSS (see image below) as well as by wilcoxsign_test in R. (See also my related question: teststatistic vs linearstatistic in wilcoxsign_test)

Others suggest the Bravais-Pearson ($r = \frac{cov(XY)}{sd(X) \times sd(Y)}$) or Spearman ($r_S$) correlation coefficients (depending on data type).

When you calculate them, the two rs are not even remotely the same. E.g., for my current data:

r = 0.23   ( for $r = \frac{Z}{\sqrt{n_x + n_y}}$ )

r = 0.43   ( Pearson )

These would imply quite different effect sizes.

So which is the correct effect size to use, and how do the two rs relate to each other?

---

Pages 224 (bottom part) and 225 from Pallant, J. (2007). SPSS Survival Manual:

Answer 1

• If you don't have ties, I would report the proportion of after values that are less than the corresponding before values.
• If you do have ties, you could report the proportion of after values that are less than before out of the total number of non-tied pairs, or report all three proportions (<, =, >) and perhaps the sum of whichever two were more meaningful. For example, you could say '33% had less fear of statistics, 57% were unchanged, and 10% had more fear after the course such that 90% were the same as or better than before'.

Generally speaking, a hypothesis test will output a p-value that can be used to make a decision about whether or not to reject the null hypothesis while controlling for the type I error rate. The p-value, however, conflates the size of the effect with our amount of clarity that it is inconsistent with the null (in essence, how much data the test had access to). An effect size generally tries to extract the $N$ so as to isolate the magnitude of the effect. That line of reasoning illuminates the rationale behind dividing $z$ by $\sqrt N$. However, a major consideration with effect size measures is interpretability. Most commonly that consideration plays out in choosing between a raw effect size or a standardized effect size. (I suppose we could call $z/\sqrt N$ a standardized effect size, for what that's worth.) At any rate, my guess is that reporting $z/\sqrt N$ won't give people a quick, straightforward intuition into your effect.

There is another wrinkle, though. While you want an estimate of the size of the overall effect, people typically use the Wilcoxon signed rank test with data that are only ordinal. That is, where they don't trust that the data can reliably indicate the magnitude of the shift within a student, but only that a shift occurred. That brings me to the proportion improved discussed above.

---

On the other hand, if you do trust that the values are intrinsically meaningful (e.g., you only used the signed rank test for its robustness to normality and outliers), you could just use a raw mean or median difference, or the standardized mean difference as a measure of effect.

Answer 2

Without knowing what kind of data were being assessed it's very hard to give good advice here. And really, that's all you can get. There's just no such thing as a best measure of effect size for questions like this... maybe ever.

The effect sizes mentioned in the question are all standardized effect sizes. But it's entirely possible that the means or medians of the original measures are just fine. For example, if you're measuring how long it takes for a manufacturing process to complete then the difference in times should be a perfectly reasonable effect size. Any changes in process, future measurements, measurements across systems, and measurements across factories, will all be in time. Maybe you want the mean or maybe you want the median, or even mode, but the first thing you need to do is look at the actual measurement scale and see if the effect size there is reasonable to interpret and strongly connected to the measure.

To assist in thinking about that, effects that should be standardized are things that are measured more indirectly and in many ways. For example, psychological scales can vary over time and in many ways and attempt to get at an underlying variable that is not directly being assessed. In those cases you want standardized effect sizes.

With standardized effect sizes the critical issue is not just which to use but what they mean. As you imply in your question, you also don't know what they mean and that's the critical thing. If you don't know what the standardized effect is then you can't report it correctly, interpret it correctly, or use it correctly. Further, if there are a variety of ways you'd like to discuss the data there is absolutely nothing stopping you from reporting more than one effect size. You can discuss your data in terms of linear relationship, like with product moment correlation, or in terms of relationship between the ranks with Spearman r and differences between those or just provide all the info in the table. There's nothing wrong with that at all. But more than anything you're going to have to decide what you want your results to mean. That's something that can't be answered from the information given and might require far more info and domain specific knowledge than is reasonable for a question in this kind of forum.

And always think meta-analytically about how you're reporting effects. Will people in the future be able to take the results I'm reporting and integrate them with others? Perhaps there's a standard in your field for these things. Perhaps you selected a non-parametric test primarily because you don't trust the conclusions others have made about underlying distributions and you want to be more conservative in your assumptions in a field that primarily uses parametric tests. In that case there's nothing wrong with additionally providing an effect size typically used with the parametric tests. These and many other issues need to be considered when thinking about how you place your finding in a larger literature of similar research. Typically good descriptive stats solve these problems.

So that's the primary advice. I have a few additional comments. If you want your effect size to be strongly related to the test you did then the Z based recommendation is obviously best. Your standardized effect size will mean the same thing as the test. But as soon as you're not doing that then there's nothing wrong with using most anything else, even something like Cohen's d that's associated with parametric tests. There is no assumption of normality for calculating means, standard deviations, or d scores. In fact, there are weaker assumptions than for the recommended correlation coefficient. And always report good descriptive measures. Again, descriptive measures have no assumptions you'd be violating but keep in mind their substantive meaning. You report descriptive stats that say something about your data you want to say and means and medians say different things.

If you want to discuss repeated measures versus independent design effect sizes then that's really a whole new question.