# Determining the statistical significance of changes to Likert-type data in multiple surveys

I have the results from two surveys taken approximately one year apart. All of the questions are Likert-type; many questions are the same between the two surveys (these are the ones in which I am interested). Some of the survey respondents may or may not be the same (the results are anonymous). For the first survey, I have some 180 responses; for the second, 71.

What is the best way to compare responses to the questions that are repeated between surveys? I'm really looking to answer questions like this: Question (number) had 14% of the respondents score 1 in Year A (i.e., 25 out of 180), but 20% (14 out of 71) in Year B. Is there a real difference between those two scores, or is this difference not statistically significant?

Hierarchical ordered logistic regression. Example of implementation in Stata https://www.stata.com/features/overview/multilevel-ordered-logistic-models/

• Excellent - I assume there are ways to do this in R? Do you happen to have any references handy? Commented Oct 6, 2022 at 18:55
• stats.stackexchange.com/questions/238581/… Commented Oct 6, 2022 at 21:12
• Big picture is that you model an overall response to your survey item and time-soecific deviations from the mean (i.e. the differences from that mean for the 2 time periods of your data). If you put your data in a long format where each row is a response to a at time T and fit the model to the question asked at 2 points in time, you can do something like lmer(response ~ (1| time)). To accommodate the ordinal response, you'll want the cumulative logistic or probit distribution. If this is over your head, you can always just fit a linear regression to that data with a dummy variable for time. Commented Oct 6, 2022 at 21:15

As @socialscientist notes, you could use some sort of hierarchical/multilevel ordinal regression model (e.g., see Bürkner & Vouree, 2019 for more information). Though hierarchical models can be somewhat complex, and you may not be familiar with them. So you could also try estimating the polychoric correlation for each item given to both sets of survey takers. The polychoric correlation is a correlation commonly used with ordinal data, and there are many good articles on the topic (e.g., Uebersax, 2015).

Regarding R packages, see brms for ordinal regression, and polychor for polychoric correlations.

References

Bürkner, P. C., & Vuorre, M. (2019). Ordinal regression models in psychology: A tutorial. Advances in Methods and Practices in Psychological Science, 2(1), 77-101.

Uebersax, J. S. (2015). Introduction to the tetrachoric and polychoric correlation coefficients. URL: http://www. john-uebersax. com/stat/tetra. htm (visited on 12/18/2014).