Seeking suggestions for a non-parametric version of two-way ANOVA/multi-variable MANOVA with repeated measures

Question

I've been reading around this and keep coming up short so I'm hopeful there's an easy test for this.

I am measuring the differences (on a Likert scale) between matched (paired) pre- and post-lesson surveys. Two sets of four questions in the survey are grouped around themes (i.e., "knowledge" or "attitude") so I need to analyze those together to measure changes rather than individually. I may also do smaller subsets of questions (i.e., two variable, hence two-way ANOVA)

For non-parametric tests, nothing seems to fit--Wilcoxon Rank Sum works for comparing single variables among pairs; Friedman works for multiple matched groups but not multiple variables. There doesn't seem to be anything for paired groups and multiple variables.

Also, our sample is around 70 and I I have read that with a large enough sample size a parametric test like ANOVA will most likely be fine. In that case a MANCOVA with repeated measures would potentially work. But I would prefer to go with the "safer" non-parametric option if there's any question about accuracy.

Thank you for any suggestions.

Answer 1

Okay, maybe I'm misunderstanding how your data are arranged but here's my attempt at an answer. This is based on the understanding that your data are arranged something like: Survey 1 will have a set of likert responses and within that there's 4 questions for each of 2 categories (knowledge and attitude). This survey is administered before the intervention and then you provide the same survey afterwards. Survey 2 asks the same questions afterwards, and you're interested in the individual's change in response, and you can control for this because you have individual identity as one of your variables?

In this case rather than attempt to run everything at once in one analysis I would break this out into individual questions that I was trying to address and use a specific test to answer each question. e.g.:

(1) Does knowledge improve after the intervention? Use a paired t-test of the average response to the knowledge questions before and after the intervention. Same idea applies for attitude.

(2) Does knowledge show a greater improvement than attitude? For each student calculate the difference between their knowledge scores between survey 1 and survey 2, then calculate the difference between their attitude scores between survey 1 and survey 2. Then perform a paired t-test to determine whether knowledge or attitude showed the greatest gains while controlling for individual identity.

(3) Is an increase in knowledge correlated with an increase in attitude? Same test as above but you're running a spearman rank correlation instead (or Wilcoxon), where delta knowledge is on the y-axis, and delta attitude is on the x-axis, and each point is an individual's average score.

This is just my opinion, but that's because I don't see the utility of lumping all of these analyses together into one assessment that's more difficult to understand.

Also, don't commit yourself to a non-parametric test just because it's "safer". Run the tests for normality so that you meet the assumptions of whatever test you're doing and choose the appropriate (parametric/non-parametric) on that basis.

Answer 2

Even when you combine your multiple Likert items into Likert scales after the items/questions "are grouped around themes," you are unlikely to have data that can be considered continuous interval-scaled data appropriate for classic ANOVA.

If you have ordinal data, why not use ordinal regression? As Frank Harrell explains in Chapter 13 of Regression Modeling Strategies, a proportional-odds logistic regression is a "generalization of Wilcoxon-Mann-Whitney-Kruskal-Wallis-Spearman" that allows for incorporating more complex designs with covariates.

If all you have for each individual is 1 pre-lession set of answers and 1 post-session set of answers, then it seems that you could model the post-pre differences in your Likert scales, in a generalization of a paired t-test. You could include the theme as a predictor of the Likert-scale post-pre differences, allowing for a combined analysis of all data (which is generally preferable).

One way to think about the within-individual correlations among responses to the various themes is to recognize that corresponding multivariate (in the sense of multiple-outcome) ANOVA designs end up with the same point estimates as for modeling each outcome separately; it's the standard errors of the estimates that need adjustment for correlations. In your situation, after you fit the proportional-odds model, you could generate a corrected variance-covariance matrix for the coefficient estimates by bootstrap resampling of the data by individual and using the empirical variance-covariance matrix of coefficient estimates over the multiple resamples.