0
$\begingroup$

I have a number of non-significant results from a Kruskal-Wallis test and wondering the best way to report them.

My independent variable is 3 categories:

Group A: Uses product A; Group B: Uses product B; Group C: Does not use either product.

The dependent variables are ordinal/Likert agreement scale.

Would it make sense to say while no statistical difference was found between Group A, Group B, and Group C, in many cases, Group A and Group B had higher mean ranks than Group C? For example, could I say something like: "A comparison of mean rank scores demonstrates that one of the two types of product users had higher mean rank scores than non-users on the following statements (S1, S2, S3...)?

$\endgroup$
  • 1
    $\begingroup$ What message are you trying to convey? That you think the reader should attach any meaning to differences you have declared are not significant? $\endgroup$ – whuber Feb 27 at 19:58
  • 1
    $\begingroup$ Thank you for posing the question. I can see how providing this information wouldn't be particularly useful to readers. Many thanks! $\endgroup$ – user3424836 Feb 27 at 20:02
  • $\begingroup$ You might want to look at an effect size statistic that will be easy to interpret by your audience. Vargha and Delaney's A reports the probability of an observation in group 2 being greater than an observation in group 1. $\endgroup$ – Sal Mangiafico Feb 28 at 0:39
  • $\begingroup$ Thank you! I found your explanation of Vargha and Delaney's A on your excellent website. I am using Stata, so wondering if is there any way to run this test on Stata? I'm also still wondering: if I report the effect size for non-significant results, is that information even useful to include? Many thanks! $\endgroup$ – user3424836 Feb 28 at 15:29
  • $\begingroup$ VDA is used for the two-sample case, which I think would make sense for what you have asked. VDA is easy to calculate from the U or W statistic from a Mann-Whitney or Wilcoxon test. Instructions are in this document. P = U/(n1 * n2) and U = W - n.smaller * (n.smaller +1) / 2. If your data aren't too unwieldy, you could also run the following R code on this site. A = c(1,2,3,4,5,6,7,8,10); B = c(5,6,7,8,9,10); library(effsize); VD.A(B, A) $\endgroup$ – Sal Mangiafico Mar 1 at 14:40
0
$\begingroup$

Answer from my comments.

One approach might be to report an effect size statistic. Vargha and Delaney's A reports the probability of an observation in one group being greater than an observation in the other group. I think this will be relatively easy for your audience to interpret. For more rigor, a confidence interval for the statistic can be reported. VDA is used for two groups.

VDA is easy to calculate from the U or W statistic from a Mann-Whitney or Wilcoxon test. Instructions are in this document.

P = U/(n1 * n2)
U = W - n.smaller * (n.smaller +1) / 2. 

(Note: Oddly, it appears that the "W" that R reports is the same as U.)

If your data aren't too unwieldy, you could also run the following R code on this site or in R.

if(!require(effsize)){install.packages("effsize")}
A = c(1,2,3,4,5,6,7,8,10)
B = c(5,6,7,8,9,10)
library(effsize)
VD.A(B, A)

   ### Vargha and Delaney A
   ###
   ### A estimate: 0.75 (large)

Note that the effsize package will return a confidence interval for Cliff's delta, which is linearly related to VDA. Package documentation is here.

cliff.delta(B, A)

    ### Cliff's Delta
    ### 
    ### delta estimate: 0.5 (large)
    ### 95 percent confidence interval:
    ###      lower     upper 
    ### -0.1213262 0.8398131

VDA = Cliff's delta / 2 + 0.5

So, converting the output for Cliff's delta, VDA = 0.75 (0.439, 0.920)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.