4
$\begingroup$

For my dissertation, I am currently evaluating pharmacy students' perception on two distinct academic electronic health record systems - A and B. The students were asked to identify how much they agreed to a set of statements via a 5 point Likert scale (Strongly disagree=1; Disagree=2; Neither agree nor disagree=3; Agree=4; Strongly Agree=5 ) to judge how much they agreed with each statement.

This survey was first filled for system A and then for system B. Now, I have to analyse the Likert scale data of both the samples but I am a bit confused on how I should proceed and everything I have read online has not been of much help.

So far, I was able to find weighted mean and SD. I saw online that I can conduct a t-test or but I do not know how my data should look like... let alone how I should even go about analysing the data.

It should be mentioned that survey A had 30 participants (which were students in Junior year), while survey B had 25 participants (which were students in Senior year). In other words, I am comparing between two different populations, but the set of statements are essentially the same for both surveys.

I just want to compare the results and show that 'X' electronic health record is revealed to be much more useable for pharmacy students.

$\endgroup$
7
  • 1
    $\begingroup$ Can you edit your post to include your data? $\endgroup$ Commented Mar 13 at 7:55
  • 1
    $\begingroup$ en.wikipedia.org/wiki/Ordinal_regression $\endgroup$ Commented Mar 13 at 7:56
  • 2
    $\begingroup$ You may want to chime in at the meta question I just opened: stats.meta.stackexchange.com/q/6653/1352 $\endgroup$ Commented Mar 13 at 9:03
  • 1
    $\begingroup$ It should be noted that the OP mentions "a set of statements" but the answers are about comparing means of responses on a single question (the same question) in two groups. I suspect that part of the OP's troubles is how to use the differences between the two groups across an array of questions. $\endgroup$
    – Pere
    Commented Mar 13 at 18:50
  • 2
    $\begingroup$ "I just want to [...] show that [...]" - I hope you mean you want to show if. $\endgroup$ Commented Mar 14 at 9:45

3 Answers 3

11
$\begingroup$

First, your dissertation committee has done you a disservice. You should have been required to figure this out before you gathered data.

Second, I don't think this is possible. Your two sets of respondents differ in a key attribute -- juniors vs. seniors -- and it won't be possible to disentangle the effect of an additional year of schooling from any differences in usability -- better trained students should find most systems more usable.

Third, if the two surveys are only "essentially" the same, then that is likely to cause additional problems.

I hope this isn't a key part of your dissertation!

$\endgroup$
7
$\begingroup$

While I agree with all three points Peter Flom makes, this will not help you in your thesis.

What you can do is to simply run an unpaired t-test against your two sets of Likert ratings, which tests against the null hypothesis of equal means. The t-test requires a normal distribution of test statistics, which is given if your two separate datasets are each normally distributed. Likert data are not normal. Fortunately, 25 or 30 data points should be enough to rely on standard asymptotic theory. This is related: t-test when the data population is not normally distributed

Alternatively, you could run a permutation or a bootstrap test for different null hypotheses - for instance, for equality of means. Good's book has some nice examples for such comparisons in its initial chapters. I personally suspect that any permutation or bootstrap test for equality of means will yield very similar results to a simple t-test.

However, whatever you do, you should take Peter Flom's points to heart and devote a large part of your thesis to the limitations your data pose: even if you do find differences, you can in no way claim that this is due to the intervention; they may just come from the differences in your study participants. There is simply very little to learn scientifically from a situation like this.

$\endgroup$
2
  • $\begingroup$ There's more than one item; & the reference to a Likert scale ought to imply, though might not, that the plan was to compare the average of the responses across items between the two groups. $\endgroup$ Commented Mar 13 at 17:05
  • $\begingroup$ Just as a reminder, I think the OP is saying that the list of survey questions is only "essentially" the same. I sort of have a feeling that essentially means not in this context. Do the surveys even cover the same content :-/ $\endgroup$
    – Weiwen Ng
    Commented Mar 13 at 18:36
6
$\begingroup$

I agree with Peter Flom's answer and Stephan Kolassa's answer to a substantial extent. Like Stephan, I think a highly-limited analysis could be performed, but I want to give my two cents on the approach I would take.

Similarly to Jarle Tufto's comment, I would take an ordinal regression approach.

Further, I would go with mixed effects on my parameters with the random effects being for the two surveys/groups. This approach at least will give better statistical precision for the fixed effects.

I would follow an exploratory Bayesian workflow that explores different models iteratively and thoughtfully. Although fanciful and hopeful in your case, it might allow you to find provisional evidence for latent structures, such as latent mixtures of parameters. I wouldn't hold my breath, but finding such a thing in an exploratory analysis might suggest future directions from a more confirmatory approach.

I recommend studying McElreath's lecture Statistical Rethinking 2023 - 11 - Ordered Categories if you plan on pursuing such an analysis. Not only does he walk through a Bayesian approach to fitting an ordinal regression on survey data, but you can also see his emphasis on causal assumptions.

Developing a causal model for your system of study will help you understand, analyze, and explain the flaws in your statistical design. I expect that you are conditioning on a collider, and I am doubtful that sensitivity analysis to confounds will help you.

$\endgroup$
2
  • 1
    $\begingroup$ I agree that this should be treated as a mixed effects model and the random effect is the year of the participants. Unfortunately this random effect is nested within the fixed effect and this is a situation like $n=1$. A Bayesian approach might work especially if there is data from literature about what sort of variation to expect. But the information from the experiment/data is very low and one may risk over-interpretation of the results by subjectively selecting priors after having seen the results. $\endgroup$ Commented Mar 15 at 7:36
  • $\begingroup$ @SextusEmpiricus Yes, all good concerns to have (+1). In Bayesian workflow it is common to fit models in an experimental fashion as a way of learning what is important; essentially a form of sensitivity analysis. I think it would be a good idea to try out a variety of distinct priors to see how sensitive the resulting posterior is to the choice of priors. $\endgroup$
    – Galen
    Commented Mar 15 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.