How do I compare the results between two different Likert Scales questionnaires?

Question

For my dissertation, I am currently evaluating pharmacy students' perception on two distinct academic electronic health record systems - A and B. The students were asked to identify how much they agreed to a set of statements via a 5 point Likert scale (Strongly disagree=1; Disagree=2; Neither agree nor disagree=3; Agree=4; Strongly Agree=5 ) to judge how much they agreed with each statement.

This survey was first filled for system A and then for system B. Now, I have to analyse the Likert scale data of both the samples but I am a bit confused on how I should proceed and everything I have read online has not been of much help.

So far, I was able to find weighted mean and SD. I saw online that I can conduct a t-test or but I do not know how my data should look like... let alone how I should even go about analysing the data.

It should be mentioned that survey A had 30 participants (which were students in Junior year), while survey B had 25 participants (which were students in Senior year). In other words, I am comparing between two different populations, but the set of statements are essentially the same for both surveys.

I just want to compare the results and show that 'X' electronic health record is revealed to be much more useable for pharmacy students.

Answer 1

First, your dissertation committee has done you a disservice. You should have been required to figure this out before you gathered data.

Second, I don't think this is possible. Your two sets of respondents differ in a key attribute -- juniors vs. seniors -- and it won't be possible to disentangle the effect of an additional year of schooling from any differences in usability -- better trained students should find most systems more usable.

Third, if the two surveys are only "essentially" the same, then that is likely to cause additional problems.

I hope this isn't a key part of your dissertation!

Answer 2

While I agree with all three points Peter Flom makes, this will not help you in your thesis.

What you can do is to simply run an unpaired t-test against your two sets of Likert ratings, which tests against the null hypothesis of equal means. The t-test requires a normal distribution of test statistics, which is given if your two separate datasets are each normally distributed. Likert data are not normal. Fortunately, 25 or 30 data points should be enough to rely on standard asymptotic theory. This is related: t-test when the data population is not normally distributed

Alternatively, you could run a permutation or a bootstrap test for different null hypotheses - for instance, for equality of means. Good's book has some nice examples for such comparisons in its initial chapters. I personally suspect that any permutation or bootstrap test for equality of means will yield very similar results to a simple t-test.

However, whatever you do, you should take Peter Flom's points to heart and devote a large part of your thesis to the limitations your data pose: even if you do find differences, you can in no way claim that this is due to the intervention; they may just come from the differences in your study participants. There is simply very little to learn scientifically from a situation like this.

Answer 3

I agree with Peter Flom's answer and Stephan Kolassa's answer to a substantial extent. Like Stephan, I think a highly-limited analysis could be performed, but I want to give my two cents on the approach I would take.

Similarly to Jarle Tufto's comment, I would take an ordinal regression approach.

Further, I would go with mixed effects on my parameters with the random effects being for the two surveys/groups. This approach at least will give better statistical precision for the fixed effects.

I would follow an exploratory Bayesian workflow that explores different models iteratively and thoughtfully. Although fanciful and hopeful in your case, it might allow you to find provisional evidence for latent structures, such as latent mixtures of parameters. I wouldn't hold my breath, but finding such a thing in an exploratory analysis might suggest future directions from a more confirmatory approach.

I recommend studying McElreath's lecture Statistical Rethinking 2023 - 11 - Ordered Categories if you plan on pursuing such an analysis. Not only does he walk through a Bayesian approach to fitting an ordinal regression on survey data, but you can also see his emphasis on causal assumptions.

Developing a causal model for your system of study will help you understand, analyze, and explain the flaws in your statistical design. I expect that you are conditioning on a collider, and I am doubtful that sensitivity analysis to confounds will help you.