1
$\begingroup$

I have a likert scale developed from a pilot survey, an EFA, and Cronbach's alpha. The survey questions are on an agree/disagree 5-point scale.

We are reporting the index score - calculated as the average individual-item mean of the 6-item index for each participant, so an average of averages.

However, I also want to inform my stakeholders on how to improve their index score. Would the most helpful form of reporting at the individual item-level for the index be: the mean of each individual item, a distribution of responses for each item, a distribution of score groupings to show the percent top-box and the percent bottom-box, a median, a mode?

What is the reasoning?

Results reporting example image attached.

Example of results reporting

$\endgroup$
0
$\begingroup$

Importance of sample size. Crucially, you do not reveal your sample sizes. Proportions and averages can be quite meaningless without them. As an extreme example, 4 heads in 10 tosses of a coin is hardly evidence of a biased coin, but 400 heads in 1000 tosses is pretty good evidence of bias.

Simulated data for illustration. Here is an experiment with simulated data with two groups of 1000 subjects each. Specifically, let's look at a 2-sample t test of differences in 'average' Likert scores and a 2-sample test whether two proportions in favor differ significantly.

Here are two groups of Likert scores sampled from somewhat different populations.

set.seed(2020) 
x1 = sample(1:5, 1000, rep=T, p=c(4,6,20,40,30))
x2 = sample(1:5, 1000, rep=T, p=c(5,5,15,30,45))

Pretending that Likert scores are numerical, we get a mean of 3.9 in one group of 1000 and and a mean of 4.1 in a second group of 1000.

summary(x1); sd(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    1.0     3.0     4.0     3.9     5.0     5.0 
[1] 1.042635  # sample SD


summary(x2); sd(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  1.000   4.000   4.000   4.064   5.000   5.000 
[1] 1.11855  # sample SD

Proportions in the two 'favorable' categories are about 72% and 76%, respectively.

n1 = sum(x1 > 3); n2 = sum(x2 > 3)
n1; n2
[1] 717
[1] 758

Welch t test. A Welch 2-sample t test (which does not assume equal variances in the two populations) to test whether the two sample means differ significantly shows a highly significant difference (P-value about $0.001 << 0.05).$

t.test(x ~ g)

        Welch Two Sample t-test

data:  x by g
t = -3.3916, df = 1988.2, p-value = 0.0007086
alternative hypothesis: 
   true difference in means is not equal to 0
95 percent confidence interval:
 -0.25883252 -0.06916748
sample estimates:
mean in group 1 mean in group 2 
          3.900           4.064 

Test of binomial proportions. A test whether the two proportions 'in favor' differ significantly (P-value $0.037 < 0.05).$ [With samples as large as these, no continuity correction is needed. The test assumes binomial proportions are approximately normally distributed, which they are for samples as large as 1000.]

prop.test(c(n1,n2),c(1000,1000), cor=F)

        2-sample test for equality of proportions 
        without continuity correction

data:  c(n1, n2) out of c(1000, 1000)
X-squared = 4.3416, df = 1, p-value = 0.03719
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.079524487 -0.002475513
sample estimates:
prop 1 prop 2 
 0.717  0.758 

Which interpretation of data is best in your situation? So, for my simulated data, looking at Likert means seems to give very strong evidence that the two groups differ and looking at proportions 'in favor' gives significance at the 5% level.

Do Likert data reliably behave as interval data? You may be surprised that I strongly prefer looking at the proportions 'in favor'. These proportions are unquestionable and direct results of the ordinal Likert responses. By contrast, the 'means' require interpretation of the Likert scores as if they were numbers, which they are not. They are ordinal categories. Whether two 'in favor' precisely balances one 'strongly opposed' can be a topic of controversial discussion.

Over the long run, I think one does better to analyze exactly what the data say, rather than what they may confess to after coercion.

Wilcoxon signed-rank test. I am not saying that looking at proportions 'in favor' is the very best way for you to look at your data.

A 2-sample nonparametric Wilcoxon rank sum test treats the data as ordinal categories. Roughly speaking, it may be taken as a test whether scores in one group may be 'shifted' relative to those in the other group. (Some people take this as a test whether the two sample medians are significantly different. For some data this is OK.)

Originally, the Wilcoxon test was not formulated for use with data having many ties (as yours do), but for large samples, the version of this test in R accommodates to ties without error messages. The (highly significant) results of this test are shown below. I think it may make good sense to use results of the Wilcoxon rank sum test because the test treats the data as the categorical categories they actually are.

This test asks and answers a different question than does the test of proportions. The issue is which question is an appropriate one for your data.

wilcox.test(x ~ g, cor=F)

        Wilcoxon rank sum test

data:  x by g
W = 439580, p-value = 7.483e-07
alternative hypothesis: 
  true location shift is not equal to 0
| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.