Which test is appropriate for comparing Pre and Post intervention data?

Question

I am conducting research in which employee engagement will be evaluated on approximately 20 participants. They will complete an 11 question Likert-scale survey, have interventions done on them (read a book, read 3 articles, and sit through a lecture), and complete the same 11 question survey. Is the correlated t-test the appropriate test to conduct on the pre and post data to determine if a statistically significant increase in employee engagement occurred?

Answer 1

Probably you should use a non-parametric alternative to the paired t-test since your sample appears very small for the underlying distributional assumptions of t-test to hold. Try the Wilcoxon signed rank test instead.

Answer 2

If there are three groups involved (read a book, read 3 articles, sit through a lecture), then a non-parametric ANOVA (Kruskal-Wallis) should be done first to determine if "a statistically significant increase in employee engagement occurred." Then, follow-up non-parametric tests can be run.

Answer 3

It is not clear whether you have 3 groups (1 treatment for each group) or just one group (that receives all treatments) that you want to compare before vs. after.

I will assume the later (please correct us if otherwise). The t-test approach assumes that the difference between an answer of 1 and an answer of 2 is exactly the same as a difference between an answer of 3 and an answer of 4, etc. This is not really a reasonable assumption (but many people do it anyways because it is easy).

Probably better for comparing before/after data with a Likert scale is McNemar's test and its generalizations.

But note that inference in these types of studies are always tricky. The assumptions of the test is that the only differences between the before scores and the after scores are the effect of the treatment and random fluctuation. But other factors could very well play a part here including test fatigue (I am tired of answering questions, so I will just put the same answer down on everything for the post test without actually thinking about it) or people trying to meet expectations (the lecture said I should be engaged, so I will choose what I think are the right answers rather than how I really feel). Even the fact that your subjects are hungrier, happier, or more tired during one evaluation than the other could have more effect than the actual treatment. There is no statistical test that can over come these limitations (there are some alternate study designs that may help some).