Repeated measures analysis of Likert-style question I have a Likert-style survey that includes 2 questions: perceived understanding before an intervention and perceived understanding after the intervention. I would like to use some form of inferential statistics to determine if there is a significant improvement in understanding post-intervention.
Issues: Likert-like questions are ordinal (not continuous); and because the same person is answering both questions, the groups (before and after) are not independent.
What is the correct test to use here?
I've seen lots of suggestions for Likert tests (Wilcoxon signed-rank test, t-test, ANOVA, Pearson correlation, bootstrapping, mixed modelling, etc.), but no posts I've seen seem to give a reasonable answer to my simple scenario. I understand there may be disagreement on the proper approach, but can anybody point me toward an acceptable approach and briefly explain its application?
 A: Ordinal variables, considered within Stevens' typology, can be operated on in a way that preserves the action of a certain group (called an isotonic group in Stevens' 1946 paper). We will avoid this underlying theory below.
I do not believe there is a uniquely-best way to answer your question, but here is a conceptually simple approach that steers clear of sophisticated or complicated arguments.
Let's say you have your survey question denoted $x_{i,b}$ for the $i$th participant before the intervention, and for that same participant you have their response to the survey question $x_{i,a}$ after the intervention. Since the survey questions are on a Likert scale, we know that $x_{i,b} \in \{1,2,3,4,5\}$ and $x_{i,a} \in \{1,2,3,4,5\}$.
Ordinal variables have one clear property: order. This means we can compare $x_{i,b} < x_{i,a}$, $x_{i,b} > x_{i,a}$, or their non-strict analogs. You could also consider $x_{i,b} = x_{i,a}$ for that matter.
Let's say you are interested in how often the participants' survey response was higher after the intervention. You can then compute the frequency probability as
$$P(X_b < X_a) \approx \frac{\text{Count}(x_b < x_a)}{n}$$
where $\text{Count}(x_b < x_a)$ is the number of participants whose intervention survey score was higher than their pre-intervention survey score, and $n$ is the number of participants.
Sometimes it is sufficient to just consider this probability, especially if it is close to zero or one. But there is a lot of variation among people, so one might wish to bootstrap this probability to inspect a histogram and confidence interval.
The probability itself is a way of quantifying the effect of the intervention, addressing your concern about non-independence in a particular way. Because the inequalities were evaluated in a way paired by the participants, you have addressed their role in answering both the before and after questions. Since we stuck to a simple frequency of how often an inequality held among participants, we have not violated Steven's typology by fallaciously treating the Likert scales as ratio or interval variables.
