4
$\begingroup$

I conducted a Kruskal Wallis test on 4 groups of participants' responses on a Likert scale.

The likert scale consisted of 5 possible responses.

This is from results from a research project I have conducted with physiotherapists in the UK.

Example of results enclosed below: A Kruskal Wallis sum rank test revealed a statistically significance across the groups (Group one (prescribing physiotherapists n = 79; group two (physiotherapist student prescriber n =12; Group three physiotherapist non prescriber n=171 Group 4 student physiotherapist n = 35). Χ2 (3,n=297) = 8.992, ρ < 0.029 Pairwise comparison between the groups revealed the following relationships

Pairwise Comparisons of Please indicate which of the following you belong to
Sample 1-Sample 2               Test Stat     SE    Std. Test Stat  Sig.    Adj. Sig.a
PSP-physio prescriber              5.846      25.240    .232    .817    1.000
PSP-physio NP                    -35.558      24.328    -1.462  .144    .863
PSP-student physio               -36.533      27.252    -1.341  .180    1.000
physio prescriber-physio NP      -29.712      11.082    -2.681  .007    .044
physio prescriber-student physio -30.687      16.542    -1.855  .064    .381
physio NP-student physio           -.975      15.114    -.065   .949    1.000

Each row tests the null hypothesis that the Sample 1 and Sample 2 distributions are the same.

Asymptotic significances (2-sided tests) are displayed. The significance level is .050.
a. Significance values have been adjusted by the Bonferroni correction for multiple tests.
Table 5.20 Pairwise comparisons: support during training
Table 5.20 shows a statistically significant difference between physiotherapist prescribers and physiotherapist non prescribers (ρ = 0.007. adj via Bonferroni correction ρ = 0.044)

Investigation of descriptive statistics for physiotherapist prescribers and physiotherapist non prescribers respectively, revealed that for physiotherapist prescribers more physiotherapists believed that they would receive support during training 67.1% versus 55.2% of those who identified as physiotherapists non prescribers. The percentage who felt that they would not receive support was similar between the two groups: 11.4% for physiotherapist prescribers and 11.2% for physiotherapist non prescribers.

here I have accumulated the agree and strongly agree options to give the outcome of 67.1% for physiotherapist prescribers, and carried out the same for those reported that they did not receive support - disagree and strongly disagree responses.

How is there a significant difference between groups that is not supported by descriptive statistics. Is this an example of a false positive? Or am I misunderstanding the stats here - quite possible.

Taking on board feedback so far it looks as though I am getting a result that is statistically significant but not important. The fact that it is not hugely important reflects the current literature but may suggest the need for further work in particular between the two groups highlighted by the KW test. Thanks for the valuable feedback all

Improve this question
$\endgroup$
15
  • 1
    $\begingroup$ Please edit your question to format the text and the output summary for readability. I added some formatting to the table in the middle but the columns are not properly aligned so the information is hard to follow. $\endgroup$
    – dipetkov
    Commented Feb 6 at 11:33
  • $\begingroup$ And please explain what table 5.20 refers to? A paper, a textbook? $\endgroup$
    – dipetkov
    Commented Feb 6 at 11:34
  • $\begingroup$ Is this a matter of a test giving “statistical significance” yet the differences do not seem important? $\endgroup$
    – Dave
    Commented Feb 6 at 11:42
  • 1
    $\begingroup$ Why do you think that the significance is "not supported by descriptive statistics"? A 12% difference is something, isn't it? By the way there is important information missing, namely how many points/possible outcomes your Likert scale has. The relevant data for significance are the full distributions of the possible outcomes in the two groups. (If your scale has three points it looks like you have given all information if I understand things correctly, but with more points something is missing.) $\endgroup$
    – Christian Hennig
    Commented Feb 6 at 13:59
  • 1
    $\begingroup$ @Colin I'm still not sure why you believe the significance is not supported by the descriptive statistics. What kind of support would you expect? And it should be clear that (assuming you ran your analysis on the original five-point scale) you need to look at the distribution of all five outcomes in order to see whether or not this is in line with the significant result (as stated also in the answer of @EdM). Combining categories may lose some of the information that causes significance. $\endgroup$
    – Christian Hennig
    Commented Feb 6 at 15:42

1 Answer 1

Reset to default
5
$\begingroup$

The grouping together of the agree and strongly agree options in your "descriptive statistics" isn't consistent with the analysis of the 5-point Likert item. An extreme example: if all 79 physiotherapist prescribers chose strongly agree and all 171 physiotherapist non-prescribers chose agree, then your "descriptive statistics" would show no difference between the groups while the Kruskal Wallis test would show a highly significant difference.

Histograms of the numbers who chose each of the 5 levels, broken down by group, would be a better description of your data. For a simpler informal description you could use the mean Likert score by group.

The p-values here seem mostly to be a function of the numbers in the groups being compared. The "statistically significant" comparison doesn't have the largest-magnitude test statistic, but (with a total of 250 individuals in the 2 groups compared) it has the smallest standard error of all the comparisons. Whether that "statistically significant" comparison is of practical significance depends on your understanding of the subject matter.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for this - that is v helpful and something I will take away and review $\endgroup$
    – Colin
    Commented Feb 6 at 15:34
  • 1
    $\begingroup$ @Colin one more thought: the Bonferroni multiple comparisons correction is unnecessarily conservative. It wouldn't have mattered in this particular case, but use the Holm adjustment to the Bonferroni correction for the same protection against Type I error while maintaining higher power. $\endgroup$
    – EdM
    Commented Feb 6 at 15:49
  • $\begingroup$ I didn't even know about the Holm adjustment. I have been trying to understand the appropriate use of statistics from text books and by running potential issues past the stats dept at my University. This question was one that I felt I wanted advice to relatively quickly and this has definitely been the case so huge thank you to all of the people who took the time to not only look at my question, reformat it so it made better sense and then gave me such helpful and thoughtful feedback - I really do appreciate it. $\endgroup$
    – Colin
    Commented Feb 7 at 21:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.