# Dunn's test: is it appropriate to use p-values to interpret relative between group similarity

For a set of data that I collected, I used the Kruskal-Wallace test to determine whether there were between-group differences in the parameter I measured. It returned a significant result, so in order to determine pairwise differences between my five treatment groups, I applied Dunn's test for multiple comparisons, which yielded these results:

I would normally use a cut off of p<0.05 to determine significance, but because of the multiple comparisons I used a Bonferroni correction. Five groups were compared (AB, Control, SM, DSV, and SW), resulting in ten pairwise comparisons. Thus I assessed significant when p was less than 0.05/10(=0.005). For the table a * indicates significance at the <0.05 level, and ** at the corrected (<0.005) level.

I can tell from the table which groups have statistically different rank means from one another (i.e. the groups are different).

My question: is it appropriate to evaluate relative between group differences based on the magnitude of the p-values? For example, is it appropriate to say, based on the data provided, that the AB-Control groups are the most similar to one another? And, Control-SM have the second highest degree of similarity out of all the comparisons made here?

• I just noticed what might be a typo: do you mean Kruskal-Wallis? Apr 30, 2019 at 23:02

From my interpretation of what you are asking, I would say no, you should not compare the magnitude of the differences based on the p-values alone.

The z scores for the comparisons are dependent on the mean differences in ranks as you say, but also on the sample size for each group (and the p-values on the degrees of freedom for your comparisons). There is a good community wiki on Dunn's test here with more info.

Therefore the p values are not just determined by the actual differences between the groups, but also the sampling effort that you put in for each treatment group. So groups for which there was less sampling effort would have less significant p-values, for the same sample mean rank difference.

I suggest you calculate confidence intervals for your mean rank differences, and compare those. This will make explicit the greater uncertainty in the size of the difference for any groups with smaller sample sizes.

Adding on Izy's answer, when presenting the data, you may show the medians for each group and organize them in ascending order. The medians should behave like the mean ranks used by the test, and are easier to be understood. You may also use superscripts in a similar way as they are used in Tukey's test to mark homogeneous subsets.

Using the medians will also help understanding where are the large differences between groups. You should also report the effect sizes, that will be a better measure of the magnitude of the differences.

• +1. Assume you mean Tukey not Tuckey? It's too small an edit so I can't make it.
– Izy
Apr 25, 2019 at 7:08
• Thanks for the heads up on the typo. Apr 25, 2019 at 13:33
• @Ertxiem thank you for the feedback. I have been looking into effect sizes, but am still a bit confused as to how to interpret my result. I applied the epsilonSquared function (from the R package rcompanion), which returned an epsilon squared value of 0.384. Given my independent variable is categorical (treatment group), and my dependent variable is a ranked response to the treatment, how can I interpret this value in the context of my experiment? Please refer me to some other page if I have missed this addressed elsewhere. Apr 30, 2019 at 22:06
• @Roxanne: the interpretation of effect sizes may change from area to area. You may use as a rule of thumb the interpretation from rcompanion.org/handbook/F_08.html Apr 30, 2019 at 23:01
• @Ertxiem Thank you! I also found this page, which has a few sentences on how to interpret the result that I found helpful. Apr 30, 2019 at 23:20

While it may not be appropriate from pure statistical theory, I think there is a pragmatic use for p-values to be employed in such a way. If we are honest, this is how many readers would naturally interpret such p-values anyway as they do describe a magnitude of some description. I have read a number of articles where p-values for a range of analysis have been ranked in some order (often from the most significant to the least).

The bigger question is whether p-values are an effective assessment of similarity. This maybe interpreted (or inferred/assumed) as such but in my reading similarity is better assessed using a distance measure and, strictly speaking, p-values are a probability not a distance.