Analysing pre-and-post intervention study with anonymous responses

Question

The study I am analysing is a pre-and-post intervention questionnaire of students' views before and after studying a module. The questionnaire was distributed in three different geographical locations, where the same instructor was delivering the same module at different times within one year. The scale used is a 5-point Likert scale (which I can also convert into a binary agree/disagree variable if needed).

The surveys are anonymous, so I can't match the sample. However, in some samples, all the students answered both the pre-and-post questionnaires (same $N$) and in others this was different (different $N$).

Overall I have 343 respondents for the pre-intervention questionnaire, and 378 respondents for the post-intervention questionnaire. However in two of the three locations, I have equal pre and post intervention responses, but they are anonymous: so they are not independent, but not matched. (This is a weakness of the data, and will be reported in the results).

My questions are: a) Given the sample size, would running an unpaired t-test on the full sample be appropriate? Or should I be going for a non-parametric test? b) Can the subsamples where there are equal respondents (i.e. all students answered both pre-and-post questionnaires) be analysed as non-independent non-matched samples?

Answer 1

a) Given the sample size, would running an unpaired t-test on the full sample be appropriate? Or should I be going for a non-parametric test?

The sample size is not typically considered as "small" but the concern that the variable of interest is a 5-point Likert scale remains. If an overwhelming amount of responses center on 1 of 5, the residual may not be normal, biasing the significant test. Not knowing the actual spread of the data, I'd say non-parametric test may be more appropriate.

b) Can the subsamples where there are equal respondents (i.e. all students answered both pre-and-post questionnaires) be analysed as non-independent non-matched samples?

If you have collect an array of demographics you may consider some post hoc matching, but otherwise to my knowledge there is no analysis that can adjust for the feature of pair-samples without any identifier.

I'd recommend just analyze them as if they are independent sample. This approach has one drawback: it pools the within person variations into the overall variation, usually biasing up the p-val. But if your independent tests already show a significant difference, paired tests should find the same result.

A benefit is that you can use that for all sites, the numbers do not need to be equal before and after the intervention. The fact that they have the same numbers do not guarantee they are the same people in both days, anyway. (Unless you know the attendance on both days were 100% or you know who were absent and their name.)

Answer 2

This is more of a comment, but too long, can give some starters for a more complete answer.

I will concentrate on a special case, the same number of respondents before and after, and we will assume they are the same, but the identification is missing, and neither there is some partial information for some partial pairing. Then one idea is to look at all possible pairings, and take the average.

So let the data be $y_{2i}, y_{1i}, i=1, \dotsc, n$, where the pairing implied by the indexing is arbitrary. So we want to calculate differences $$ D_\sigma = (\bar{y_2}-\bar{y_1})_\sigma = \frac1n\left( \sum_i y_{2i}- \sum_i y_{1\sigma(i)} \right) $$ where $\sigma$ is a permutation on $\{1,2,\dotsc,n\}$. Then we average that over all the $n!$ different permutations: $$ D_{\text{Av}}= \frac1{n!} \sum_\sigma D_\sigma \\ = \frac1{n!} \sum_\sigma \left\{ \frac1n \sum_i y_{2i} -\frac1n \sum_\sigma y_{2\sigma(i)} \right\} \\ = \frac1n \frac1{n!} \sum_i \sum_\sigma \\ = \frac1n \frac1{n!} \sum_i \left\{ n! y_{2i}-\sum_\sigma y_{1\sigma(i)} \right\} \\ \qquad \text{but $\sum_\sigma y_{1\sigma(i)}= n! \bar{y_1}$ so }\\ =\frac1n \frac1{n!} \sum_i \left\{ n! y_{2i} - n! \bar{y_1}\right\} \\ =\bar{y_2} - \bar{y_1}. $$ which is indeed a support for the other excellent answer. An analysis of the variance is still needed. The important thing now is that this kind of analysis could be extended to include some information about partial pairing.