# p value <0.05 Bonferroni adjusted but associated descriptive statistics not supportive of difference

I conducted a Kruskal Wallis test on 4 groups of participants' responses on a Likert scale.

The likert scale consisted of 5 possible responses.

This is from results from a research project I have conducted with physiotherapists in the UK.

Example of results enclosed below: A Kruskal Wallis sum rank test revealed a statistically significance across the groups (Group one (prescribing physiotherapists n = 79; group two (physiotherapist student prescriber n =12; Group three physiotherapist non prescriber n=171 Group 4 student physiotherapist n = 35). Χ2 (3,n=297) = 8.992, ρ < 0.029 Pairwise comparison between the groups revealed the following relationships

Pairwise Comparisons of Please indicate which of the following you belong to
Sample 1-Sample 2               Test Stat     SE    Std. Test Stat  Sig.    Adj. Sig.a
PSP-physio prescriber              5.846      25.240    .232    .817    1.000
PSP-physio NP                    -35.558      24.328    -1.462  .144    .863
PSP-student physio               -36.533      27.252    -1.341  .180    1.000
physio prescriber-physio NP      -29.712      11.082    -2.681  .007    .044
physio prescriber-student physio -30.687      16.542    -1.855  .064    .381
physio NP-student physio           -.975      15.114    -.065   .949    1.000


Each row tests the null hypothesis that the Sample 1 and Sample 2 distributions are the same.

Asymptotic significances (2-sided tests) are displayed. The significance level is .050.
a. Significance values have been adjusted by the Bonferroni correction for multiple tests.
Table 5.20 Pairwise comparisons: support during training
Table 5.20 shows a statistically significant difference between physiotherapist prescribers and physiotherapist non prescribers (ρ = 0.007. adj via Bonferroni correction ρ = 0.044)


Investigation of descriptive statistics for physiotherapist prescribers and physiotherapist non prescribers respectively, revealed that for physiotherapist prescribers more physiotherapists believed that they would receive support during training 67.1% versus 55.2% of those who identified as physiotherapists non prescribers. The percentage who felt that they would not receive support was similar between the two groups: 11.4% for physiotherapist prescribers and 11.2% for physiotherapist non prescribers.

here I have accumulated the agree and strongly agree options to give the outcome of 67.1% for physiotherapist prescribers, and carried out the same for those reported that they did not receive support - disagree and strongly disagree responses.

How is there a significant difference between groups that is not supported by descriptive statistics. Is this an example of a false positive? Or am I misunderstanding the stats here - quite possible.

Taking on board feedback so far it looks as though I am getting a result that is statistically significant but not important. The fact that it is not hugely important reflects the current literature but may suggest the need for further work in particular between the two groups highlighted by the KW test. Thanks for the valuable feedback all

• Please edit your question to format the text and the output summary for readability. I added some formatting to the table in the middle but the columns are not properly aligned so the information is hard to follow. Commented Feb 6 at 11:33
• And please explain what table 5.20 refers to? A paper, a textbook? Commented Feb 6 at 11:34
• Is this a matter of a test giving “statistical significance” yet the differences do not seem important?
– Dave
Commented Feb 6 at 11:42
• Why do you think that the significance is "not supported by descriptive statistics"? A 12% difference is something, isn't it? By the way there is important information missing, namely how many points/possible outcomes your Likert scale has. The relevant data for significance are the full distributions of the possible outcomes in the two groups. (If your scale has three points it looks like you have given all information if I understand things correctly, but with more points something is missing.) Commented Feb 6 at 13:59
• @Colin I'm still not sure why you believe the significance is not supported by the descriptive statistics. What kind of support would you expect? And it should be clear that (assuming you ran your analysis on the original five-point scale) you need to look at the distribution of all five outcomes in order to see whether or not this is in line with the significant result (as stated also in the answer of @EdM). Combining categories may lose some of the information that causes significance. Commented Feb 6 at 15:42

The grouping together of the agree and strongly agree options in your "descriptive statistics" isn't consistent with the analysis of the 5-point Likert item. An extreme example: if all 79 physiotherapist prescribers chose strongly agree and all 171 physiotherapist non-prescribers chose agree, then your "descriptive statistics" would show no difference between the groups while the Kruskal Wallis test would show a highly significant difference.