1
$\begingroup$

I have a set of employee engagement survey results scored on a Likert scale from one company. There approximately ten survey questions. Typically there are a few hundred responses per company.

I have the same survey results from a number of other companies (Approximately ten). For each question, I want to conduct a statistical test to determine whether employees are more of less engaged versus the benchmark. I am struggling with how to conduct this.

It seems like one approach would be to translate the Likert scale to numerical values. For example, perhaps strongly agree =5, agree=4, neutral=3, etc. Then I believe I could perform an unpaired t-test assuming unequal variance to test whether the mean engagement level is different than the mean engagement level for the benchmark group.

Second approach I think of would be to calculate a percentage favorable (defined as the percentage of people that selected strongly agree or agree) and perform a test for a difference in proportion. I understand that I could use a t-test here again.

Third approach would be the same as the second approach, but instead of running the test on the percentages, I could translate the percentage favorable into count of favorable to create a 2x2 table of favorable vs not favorable and test company vs benchmark.Then I could use a Chi-squared table.

What is the recommended approach here?

Any help is greatly appreciated!

$\endgroup$
3
  • $\begingroup$ Those are reasonable ideas (though I would scratch the second one because engagement proportions could be quite negatively biased, I would favor the chi square or Fisher's exact instead) $\endgroup$ Aug 22, 2019 at 21:21
  • $\begingroup$ There probably are several useful approaches. But I would need more info to give a recommendation. Take Q1 for example. How many responses do you have from the 'benchmark' company With what are you comparing these Q1 responses? One other company (at a time) or Q1 scores from 'a number` of other companies merged together? How many other companies? How many questions on the questionnaire? // I suppose you know that if you do tests for 1000 comparisons at the 5% level, you will find 5%-significant differences in 50 comparisons, even if no differences exist. $\endgroup$
    – BruceET
    Aug 22, 2019 at 21:28
  • $\begingroup$ Thanks both! @BruceET, I have attempted to update my question to address your questions. This is very useful. Thank you so much. $\endgroup$
    – ML_Dev
    Aug 22, 2019 at 21:40

1 Answer 1

2
$\begingroup$

Here are fake data for Q1 for illustration at the benchmark company (200 responses) and at one of the other companies (352). The p vectors indicate proportions; R turns them into probabilities. The other company is simulated to have more Likert 2s and 3's and fewer Likert 5's than the benchmark company.

set.seed(2019)
bnchmrk = sample(1:5, 200, rep=T, p=c(1,1,2,3,5))
co.nr.2 = sample(1:5, 352, rep=T, p=c(1,2,3,3,2))
DTA = rbind(tabulate(bnchmrk),tabulate(co.nr.2)); DTA

Here is a chi-squared table of counts for various Likert categories:

DTA
     [,1] [,2] [,3] [,4] [,5]
[1,]   15   13   30   47   95   # benchmark
[2,]   28   60  101  100   63   # other co

Chi-squared test of homogeneity. The chisq.test procedure shows a highly significant difference in the pattern of Likert responses:

chi.out = chisq.test(DTA); chi.out

        Pearson's Chi-squared test

data:  DTA
X-squared = 61.034, df = 4, p-value = 1.759e-12

A matrix of observed counts is retrieved usind $-notation to check against the original data. A matrix of expected counts shows that they are all greater than 5.

chi.out$obs
     [,1] [,2] [,3] [,4] [,5]
[1,]   15   13   30   47   95
[2,]   28   60  101  100   63
chi.out$exp
         [,1]     [,2]     [,3]     [,4]      [,5]
 [1,] 15.57971 26.44928 47.46377 53.26087  57.24638
 [2,] 27.42029 46.55072 83.53623 93.73913 100.75362

Because there are significant differences, Pearson residuals with large absolute values point the way to the largest discrepancies between observed and expected counts. Differences in Likert scores 2, 3, and 5 seem worth examining.

chi.out$res
           [,1]      [,2]      [,3]       [,4]      [,5]
[1,] -0.1468694 -2.615122 -2.534877 -0.8578879  4.989817
[2,]  0.1107069  1.971223  1.910735  0.6466573 -3.761216

Two-sample tests. A Welch two sample t test and a nonparametric 2-sample Wilcoxon (rank sum) test both show highly significant differences between the two companies.

The t test makes sense only if you are willing to view Likert scores as numerical data for which averages make sense. The accompanying one-sided CI shows that Likert scores at the benchmark company average about half a point higher.

I did one-sided tests, which would be appropriate if you hypothesized before seeing the data that the benchmark company would have higher Likert scores.

t.test(bnchmrk, co.nr.2, alt="gr")

        Welch Two Sample t-test

data:  bnchmrk and co.nr.2
t = 6.0534, df = 394.07, p-value = 1.653e-09
alternative hypothesis: 
  true difference in means is greater than 0
95 percent confidence interval:
 0.4784205       Inf
sample estimates:
mean of x mean of y 
   3.9700    3.3125 

wilcox.test(bnchmrk, co.nr.2, alt="gr") 

        Wilcoxon rank sum test 
     with continuity correction

data:  bnchmrk and co.nr.2
W = 46800, p-value = 1.605e-11
alternative hypothesis: 
   true location shift is greater than 0

On request (with parameter conf.int=T) the Wilcoxon test also provides a confidence interval (not shown here).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.