Convert Cohen's d to nonparametric effect size and use as equivalence bounds

Question

In a within-subjects design (n = 30), I had planned to run equivalence tests with the equivalence bounds set at the effect size of Cohen's d 0.35. This is partly based on previous research and partly due to my plan to infer practical (and not just theoretical) significance from the analysis.

It was unforeseen that the data would have non-normal distribution of residuals, and therefore I will rely on equivalence tests using Wilcoxon signed-rank tests. Now I run into the issue of what my equivalence bounds should be.

Conversion of effect sizes

1. Does it make sense to convert my intended Cohen's d into Wilcoxon r or rank biserial-correlation (rrb)? Naively, I thought I could find a midpoint between "small" and "medium" for either Wilcoxon r or rrb, but I've just realized that the size interpretation heuristics are less well-defined for non-parametric effect sizes. What's a valid way to convert from d to r or rrb?

Visualisation

2. What's a good way to graph the equivalence test results from Wilcoxon signed rank tests? I don't think it's a valid approach to visually inspect whether 90% confidence intervals fall within the equivalence bounds, even if I plot the CIs for median of differences? Or would it be better practice (or clearer scientific communication) to plot the CIs in terms of rank biserial-correlation instead?

Answer 1

If the median of the differences in paired values is the statistic of interest, the one-sample sign test is the common hypothesis test that addresses this.

In this case, the confidence interval about the median of the differences makes sense to calculate.

I assume that for what you are doing, you are using the an overall median or historic median as the mu, if you will, the value to which you are comparing groups of values.

In terms of effect size, there are a couple of options you might consider.

One simple statistic is stochastic dominance. This is simply the proportion of observations (differences) greater than mu minus the proportion of observations less than mu. Simple transformations of this statistic --- but which are usually used in the unpaired two sample case --- are Vargha and Delaney's A, Cliff's delta, and the rank biserial correlation.

I'm not sure if a dominance statistic can be coerced to work for your purpose.

Another effect size statistic that makes sense is the difference between the median of differences and mu divided by the median absolute differences (MAD). I propose this statistic here for the the unpaired two sample case stats.stackexchange.com/questions/497295/effect-size-calculation-for-comparison-between-medians/.

This statistic can be used in the one-sample case by simply defining the denominator as the median of differences, minus mu. And the denominator as the MAD of the observed values (differences).

This statistic is analogous to Cohen's d in the one sample case. And if it's tested on a normal sample, and the corrected MAD is used, as in the default in R, the result will be similar to Cohen's d.

cohenD = (Mean - Mu) / Sd

mangiaficoD = (Median - Mu) / MAD

Of course, to the heart of the question, what cutoff of any of these effect sizes should you use as a cutoff analogous to a Cohen's d of 0.35 ? I don't know. But you might start with some toy data for paired differences, normally distributed, that would yield a Cohen's d of 0.35, and convert it to these different statistics. My proposed mangiaficoD = 0.35 makes sense, I think, at least when starting with a normal sample. You might have to play with other distributions and see what you think.

I don't know if any of this will result in a procedure that does what you want. It might be a case of trying some of it, and seeing if it approximates what you want. Perhaps starting with some toy data.

Here is some code in R for some of these statistics. You can run multiple iterations. If you don't use R, you can run the code here without installing software: rdrr.io/snippets/

Diff = rnorm(30, 1, 0.35)
Mu   = 0 

library(lsr)

cohensD((Diff-Mu), mu = Mu)

mangiaficoD =  median(Diff-Mu)/ mad(Diff)
mangiaficoD

Dominance = (sum(Diff>Mu) - sum(Diff<Mu))/length(Diff)
Dominance

library(DescTools)

MedianCI(Diff, conf.level = 0.95, method = "exact")

cat(median(Diff-Mu), " ", median(Diff-Mu)+mad(Diff), " ", median(Diff-Mu)-mad(Diff))

Answer 2

I will offer a possible conversion formula that you might consider. However, as you suggest, the use of this with a rank biserial correlation is unclear. Also note, that unlike the Cohen's $d$ effect size, this will have a dependence on the sample size(s).

This is based on the idea that the 2 independent samples t-test can be viewed as a point biserial correlation. As such, the relationships between t-ratios and correlations $r$ hold.

The effect size and the t-ratio are related as $$d = t · \sqrt{\frac{1}{n_1}+\frac{1}{n_2}} = t·\sqrt{\frac{n_1+n_2}{n_1n_2}}$$ Solving this for $t^2$, we obtain $$t^2=d^2·\frac{n_1n_2}{n_1+n_2}$$

Now, the relationship between correlation and t-ratios (for the 2 independent sample context) is: $$r = \sqrt{\frac{t^2}{t^2+n_1+n_2-2}}$$

Thus, $$r = \sqrt{\frac{d^2·\frac{n_1n_2}{n_1+n_2}}{d^2·\frac{n_1n_2}{n_1+n_2}+n_1+n_2-2}}$$ or "simplified" $$r = \sqrt{\frac{d^2}{d^2+\frac{(n_1+n_2-2)n_1n_2}{n_1+n_2}}}$$

Again, I provide this without any idea if it is indeed the appropriate transformation in this context.