How to compare Likert scale items before and after an intervention?

Question

I have been struggling with this question all day, reading everything I could find, but still do not have a clear answer.

A part of my research was that: 30 students tried a new educational instrument (virtual patient). Before the interaction with the patient they filled in 18 Likert items, from 1 to 5 (1=highly disagree, 5=highly agree) about their expectations about the instrument, i.e. "I am going to be deeply concentrated during the exercise". After the exercise, they filled in 18 similar Likert items, again from 1 to 5, about the actual experience, i.e. "During the exercise, i was deeply concentrated".

An interesting aspect for me, is to examine weather there is a significant difference between the answers before and after the exercise, looking at each likert item separately.

What test should I use? If I get it right, t-test would not be the case here. Or am I wrong? What about Wilcoxon signed rank-test? Should I present and compare means or medians?

As you can probably understand I am by no means an expert in statistics, and I appreciate deeply any kind of help.

Answer 1

Not exactly an answer but perhaps can help start a discussion.

To me the design of the questions seems problematic. The pre-question measures "determination" and the post-question measures "experience." Someone may be too anxious to decide to be focused, but then experienced a very focused session. I am not sure what you can get out of that. It feels like someone standardized weight and blood cholesterol into z-scores and then decided to take a difference between the two... I cannot wrap my head around this kind of operations. Perhaps other specialists in education research here can chime in.

If paired t-test is the staple of your design, then a better (hindsight) approach is to keep the questions before and after the same. e.g. "I feel confident talking to a patient," "I feel confident that I can address my patient's inquiries." etc. That way you can test if the virtual patient experience has changed any of their attitudes, beliefs or states of empowerment.

In your case, I'd rather opt for a Pearson's correlation coefficient, which would allow you to test if students with a higher determination to be focused really did focus during the process.

Answer 2

I would use a Wilcoxon signed-rank test, separately for each question.

The paired t-test is appropriate if either 1) the data are distributed normally or 2) the sample size is large enough that the central limit theorem plausibly applies. I don't think either of these things is true here. 30 might be big enough for the CLT to apply or it might not.

I was able to quickly find two on-point papers one and two. You might use those as a starting point for exploring the relevant literature (i.e. follow its references back and use a citation index to follow forward papers which reference it).

For presentation, you want to present what you learned by collecting and analyzing the data, and it's hard to tell you what to present without knowing what the data say. At a minimum, you want to present median response before, median response after, and median change in response, along with the Wilcoxon test results.

But, what if the effect of the intervention was to polarize people? The median might be 3 both before and after, but people might have migrated from mostly 3s to mostly 1s and 5s. If you are focusing on only one of the questions, you can present the full 5x5 matrix of responses. You, personally, certainly should look at all 18 of these 5x5 matrixes to try to see what is going on.