0
$\begingroup$

I've been reading around this and keep coming up short so I'm hopeful there's an easy test for this.

I am measuring the differences (on a Likert scale) between matched (paired) pre- and post-lesson surveys. Two sets of four questions in the survey are grouped around themes (i.e., "knowledge" or "attitude") so I need to analyze those together to measure changes rather than individually. I may also do smaller subsets of questions (i.e., two variable, hence two-way ANOVA)

For non-parametric tests, nothing seems to fit--Wilcoxon Rank Sum works for comparing single variables among pairs; Friedman works for multiple matched groups but not multiple variables. There doesn't seem to be anything for paired groups and multiple variables.

Also, our sample is around 70 and I I have read that with a large enough sample size a parametric test like ANOVA will most likely be fine. In that case a MANCOVA with repeated measures would potentially work. But I would prefer to go with the "safer" non-parametric option if there's any question about accuracy.

Thank you for any suggestions.

$\endgroup$
4
  • $\begingroup$ What exactly is the reason that you're not just doing a paired t-test for each set of questions? You expect there to be some sort of interaction effect between the responses under the different themes? If your concern is the multiple analyses I'd just apply a Bonferroni correction to the tests afterwards (or you could choose a less coderivative option if you were worried that would obliterate your significance). $\endgroup$
    – Dugan
    May 15, 2023 at 18:51
  • $\begingroup$ Or are you saying you want to pool responses to different questions by theme, while also controlling for individual identity? Maybe I'm misunderstanding the way your data is organized. Are the responses continuous or categorical? Are there multiple responses being pooled? Are they all pre-/post responses that you're interested in comparing? $\endgroup$
    – Dugan
    May 15, 2023 at 18:53
  • $\begingroup$ Hi Dugan, thanks for your comments. Yes, all responses are pre-/post surveys that are matched to individuals. And yes we are pooling questions by theme while controlling for individual identity. The questions within each theme area are related and probably have some interaction, which is why we want to run them together. Additionally, for us to show significant "attitude" or "knowledge" change, we need to show a net positive effect within each theme, not just with individual questions. $\endgroup$
    – MBVZ
    May 15, 2023 at 19:16
  • $\begingroup$ Oh and to answer your other question, this is all ordinal (likert) data). $\endgroup$
    – MBVZ
    May 15, 2023 at 19:47

2 Answers 2

0
$\begingroup$

Okay, maybe I'm misunderstanding how your data are arranged but here's my attempt at an answer. This is based on the understanding that your data are arranged something like: Survey 1 will have a set of likert responses and within that there's 4 questions for each of 2 categories (knowledge and attitude). This survey is administered before the intervention and then you provide the same survey afterwards. Survey 2 asks the same questions afterwards, and you're interested in the individual's change in response, and you can control for this because you have individual identity as one of your variables?

In this case rather than attempt to run everything at once in one analysis I would break this out into individual questions that I was trying to address and use a specific test to answer each question. e.g.:

(1) Does knowledge improve after the intervention? Use a paired t-test of the average response to the knowledge questions before and after the intervention. Same idea applies for attitude.

(2) Does knowledge show a greater improvement than attitude? For each student calculate the difference between their knowledge scores between survey 1 and survey 2, then calculate the difference between their attitude scores between survey 1 and survey 2. Then perform a paired t-test to determine whether knowledge or attitude showed the greatest gains while controlling for individual identity.

(3) Is an increase in knowledge correlated with an increase in attitude? Same test as above but you're running a spearman rank correlation instead (or Wilcoxon), where delta knowledge is on the y-axis, and delta attitude is on the x-axis, and each point is an individual's average score.

This is just my opinion, but that's because I don't see the utility of lumping all of these analyses together into one assessment that's more difficult to understand.

Also, don't commit yourself to a non-parametric test just because it's "safer". Run the tests for normality so that you meet the assumptions of whatever test you're doing and choose the appropriate (parametric/non-parametric) on that basis.

$\endgroup$
0
$\begingroup$

Even when you combine your multiple Likert items into Likert scales after the items/questions "are grouped around themes," you are unlikely to have data that can be considered continuous interval-scaled data appropriate for classic ANOVA.

If you have ordinal data, why not use ordinal regression? As Frank Harrell explains in Chapter 13 of Regression Modeling Strategies, a proportional-odds logistic regression is a "generalization of Wilcoxon-Mann-Whitney-Kruskal-Wallis-Spearman" that allows for incorporating more complex designs with covariates.

If all you have for each individual is 1 pre-lession set of answers and 1 post-session set of answers, then it seems that you could model the post-pre differences in your Likert scales, in a generalization of a paired t-test. You could include the theme as a predictor of the Likert-scale post-pre differences, allowing for a combined analysis of all data (which is generally preferable).

One way to think about the within-individual correlations among responses to the various themes is to recognize that corresponding multivariate (in the sense of multiple-outcome) ANOVA designs end up with the same point estimates as for modeling each outcome separately; it's the standard errors of the estimates that need adjustment for correlations. In your situation, after you fit the proportional-odds model, you could generate a corrected variance-covariance matrix for the coefficient estimates by bootstrap resampling of the data by individual and using the empirical variance-covariance matrix of coefficient estimates over the multiple resamples.

$\endgroup$
3
  • $\begingroup$ Hi Dugan and EdM, thanks for your replies--it does seem there are multiple approaches possible and I'm afraid some of your suggestion EdM is beyond what I'm probably able to do. But I understand your main point, which is that running a group test would be no different than running each question individually. Since we are mainly interested in significant change within themes from pre-to post (rather than correlation between them), would it make sense to simply add the responses per theme per individual and then run a t-test on totals for pre-post surveys? I'm thinking of simplest approaches $\endgroup$
    – MBVZ
    May 19, 2023 at 16:45
  • 1
    $\begingroup$ @MBVZ that would be taking multiple related Likert items, summing them up into a Likert scale for analysis, and treating the summed Likert scale as standard continuous "interval" data. That often is done and cam be OK; see this page for discussion. I'd recommend at least learning how to do ordinal regression, however, as it's a widely applicable approach that can work when that assumption of continuous interval data isn't tenable and you need to extend simpler non-parametric methods like Wilcoxon-Mann-Whitney. $\endgroup$
    – EdM
    May 19, 2023 at 18:25
  • $\begingroup$ Thank you for confirming--that is helpful in terms of thinking about what we can do easily with the data. will also look into OR further. $\endgroup$
    – MBVZ
    May 22, 2023 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.