I collected Likert Item responses to a series of questions. My goal is to demonstrate that some of the groups are different from the others.

The dataset has the following format:

 UserID    Group      Q1      Q2      Q3 ...
  user1    Group_A     5       4       5
  user2    Group_B     3       1       5 

Each of Q1, Q2 and Q3 is a Likert item, answered using the scale:

1: strongly disagree
2: disagree 
3: neither agree nor disagree
4: agree
5: Strongly agree 

The Group column contains 6 different groups (Group_A, Group_B, Group_C, Group_D, Group_E, Group_F).

In order to determine if there was a significant difference between any of the groups for each of the Likert Items, I used:

> kruskal.test(Q1 ~ Group, data=dt) 
Kruskal-Wallis rank sum test
data:  Q1 by Group
Kruskal-Wallis chi-squared = 148.14, df = 5, p-value < 2.2e-16

This highlights a significant difference between the groups, which may be further explored using:

> posthoc.kruskal.dunn.test(Q1 ~ group, data=dt)
Pairwise comparisons using Dunn's-test for multiple 
                     comparisons of independent samples 
data:  Q1 by Group 

                 Group_F       Group_A     Group_B       Group_C Group_D
Group_A          0.235           -           -             -                  -                   
Group_B        1.7e-14         < 2e-16     -             -                  -                   
Group_C        7.1e-05         1.2e-09     0.032         -                  -                   
Group_D        1.0e-05         8.6e-10     0.906         0.852              -                   
Group_E        9.5e-08         2.3e-13     0.906         0.726              0.990               
P value adjustment method: holm 

This helps to identify similarities/differences between groups.

Now it comes to reporting this data!

Based on this question, some suggestions are made around how to highlight a difference between two groups. However, how should the differences between multiple groups be published? Would it make sense to just state that a significant difference exists (and give the p-value from the Kruskal-Wallis test), and then somehow visually demonstrate the differences, such as a table with medians or a graph of some kind?

Any references you could recommend on this type of problem or other papers/reports that write up results similar to this?

  • $\begingroup$ Is there something wrong with the table you are presenting? How can the comparison of Group A to Group A have a p-value of 0.235 ? $\endgroup$ – Sal Mangiafico Jul 19 '17 at 21:24
  • $\begingroup$ Fixed the table - the groups were made-up. I had them in the wrong order. $\endgroup$ – Judy Jul 19 '17 at 21:29

When presenting multiple comparisons among groups, it is helpful to use a compact letter display. In this format, each group is assigned one or more letters. Groups sharing a letter are not significantly different.

Creating a compact letter display can be done by visual inspection of your table of p-values, by an algorithm by hand, or with computer code.

The package multcompLetters has a function to create the information for a compact letter display from a table of p-values. However, this function requires a full square matrix of p-values, not the compact matrix that is outputted by several R functions.

The code below uses the rcompanion package to make creating the compact letter display easier. This package relies on several other packages for things, so it may take a while for initial installation.

In all cases, when creating a compact letter display, be sure to double check the results with the original output. It is easy along the way to make a mistake manipulating the matrix of p-values or assigning letters. In general, it is best if you can arrange your matrix or output table by e.g. increasing median before creating a compact letter display to avoid spurious or weird results.

### Install packages


### Create data

Group = c(rep("Group.A", 5),rep("Group.B", 5),rep("Group.C", 5),rep("Group.D", 5))
Value = c(5,6,7,7,8,6,7,8,9,9,15,16,17,18,19,10,11,13,14,15)
Data = data.frame(Group, Value) 

### Dunn test with PMCMR


Post = posthoc.kruskal.dunn.test(Value ~ Group, data=Data)

PT = Post$p.value


### Convert matrix to full matrix


PT1 = fullPTable(PT)


### Create compact letter display information


                reversed = FALSE)

In FSA there is a function for Dunn test whose output I like better, and I think is easier to convert to a compact letter display.

### Dunn test with FSA


PT2 = dunnTest(Value ~ Group, data=Data,

PT3= PT2$res


### Create compact letter display


cldList(P.adj ~ Comparison, data=PT3,
        threshold  = 0.05)

In my opinion, it is appropriate to present Likert data with medians or other quantile-based measures. We usually do this by assigning numbers to the categories, but this is strictly for convenience. R can determine quantiles for ordered category variables.

A barplot of counts is also appropriate for ordered category data.

Answers = c("A", "A", "A", "N", "N", "SA", "SA", "D", "SD")

Q1 = factor(Answers, ordered=TRUE,
            levels=c("SD", "D", "N", "A", "SA"))


quantile(Q1, 0.25, type=1)


quantile(Q1, 0.75, type=1)



enter image description here

However, if we are using numbers to represent the categories, results could be presented as box plots.

The code below also manually adds the letters from the compact letter display information above.


  A = max(Data$Value[Data$Group=="Group.A"]) + 1
  B = max(Data$Value[Data$Group=="Group.B"]) + 1
  C = max(Data$Value[Data$Group=="Group.C"]) + 1
  D = max(Data$Value[Data$Group=="Group.D"]) + 1
  G = 0
  H = 22

  DF = data.frame(x = c(1,2,3,4), 
                  y = c(A,B,C,D), 
                  g = c("a", "a", "b","ab"))
  ggplot(Data, aes(x = Group, y = Value)) +
  geom_boxplot() +
    ylim(G,H) +

  geom_text(data = DF,
            aes(x=x,y=y,label=g)) +

  ylab("Value") +

enter image description here

  • $\begingroup$ Wow! Thank you for the detailed and thorough response. I truly appreciate you taking the time to provide such detail and all the examples. $\endgroup$ – Judy Jul 20 '17 at 14:19
  • $\begingroup$ I hope it helps. Try out the code, and if you are satisfied with the answer, you can accept the answer. $\endgroup$ – Sal Mangiafico Jul 20 '17 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.