Pre/Post Likert survey analysis

Online anonymous Likert survey was sent containing 13 items, the survey was sent before and after an intervention. I would like to analyze the % respondents who responded agree/strongly agree to the items both pre and post-intervention.

The total $$n$$ is small and do not match up as pre-intervention $$n=14$$, post-intervention $$n=18$$. I have been struggling to find the correct analysis ... I've come across McNemar vs. Wilcoxon vs paired t-test.

• "analyze the % respondents who responded agree/strongly agree to the items". This is somewhat ambiguous. Are you interested in analyzing each item individually, or counting the 13 items together? – Sal Mangiafico Feb 4 '18 at 16:59
• Having mismatched respondents pre- and post- will be problematic. Your best bet will probably be to include only those respondents who participated in both surveys. I assume you able to identify the pre- answers to the post- answers for the same respondent? – Sal Mangiafico Feb 4 '18 at 17:02
• Based on the survey design, I will not be able to match the pre and post answers for the same respondent. I am analyzing each item separately ie: For Item 1, is there a significant increase in respondents who agree and strongly agree post-intervention compared to pre-intervention. – user194003 Feb 5 '18 at 15:24

Because you can’t match the pre answers with the post answers, there’s no way to pair the responses, and so there’s no way to use tests designed for paired responses (such as those you mention: McNemar, paired-sign rank, and paired t-test).

Probably the best approach is to use a test of association designed for independent samples. Because you are measuring the same individuals in the two groups, this may be a violation of the independence assumption of the test. But I don’t know any way to adjust for this. This violation should be noted in the methods or results.

This approach will probably have lower power than if you had recorded the identity of the respondents in order to pair the results.

Because you are condensing the responses of a single Likert item into two categories (SA/A vs. N/D/SD), this becomes a nominal variable with two levels * ** .

This leaves you with a 2 x 2 contingency table. Appropriate analyses are chi-square test of association, Fisher’s exact test, or G-test of association. Please be sure you understand the assumptions and interpretation of the test you choose to use.

I also strongly suggest you report a measure of effect size. For a 2 x 2 table, the most common measure of effect size is phi. This statistic is fairly common, and may be more meaningful to the reader than the p-value in this case.

Because your observations aren't paired, you can include all observations. The unequal sample sizes won't be problematic.

** You can think of it as either an ordinal variable with two categories or a nominal variable with two categories. It won’t matter.

*** I probably recommend against condensing your data in this manner. If you leave your responses as an ordinal variable (SD, D, N, A, SA), appropriate tests might include Cochran-Armitage test, Mann-Whitney test, or Kendall correlation.

I'm sorry, but, while I agree with @Sal 's answer above, I'm even more skeptical. If you have some of the same people before and after but don't know who is who, then I'd say you on such dangerous ground that the best course may be to abandon the analysis.

It's certainly true, as Sal notes, that not being able to pair will greatly lower power. But, in addition, any results you report (beyond the simple %ages) will be misleading. Any effect size estimates, p values or other results will simply not be correct.

There are some other posts on this site about similar problems. While it would clearly be better to know the pairing, everything is not lost, so I clearly disagree with @Peter Flom.

I have one answer to Analysing pre-and-post intervention study with anonymous responses which uses permutations (in a case with exactly the same people pre/post.) That analysis should be possible to extend (I will try so, later.) Some other relevant posts is t-test for partially paired and partially unpaired data with many good answers.