Multiple non-parametric pairwise tests for small sample sizes

Question

I gave my class pre-and post-surveys on their perceptions of their own learning and engagement in my course. The survey had four sections: Confidence in Conservation Biology, Self Assessment of Skills, Self-efficenacy in Science, and Relationship to Conservation Biology. Each survey had 25 questions overall. Each question had five possible responses on a Likert scale ranging from Strongly Disagree to Strongly Agree.

Fifteen students completed the pre-survey. Only ten were present to complete the post survey.

1. Can I consider each set of the survey to be a separate study to reduce the likelihood of type-ii error, or is that not okay?

2. If I use a non-parametric test, does that reduce the chance of type-ii error?

3. Which test do you recommend I use? Is a paired test the right way to go, even though I will lose 5 data points?

Answer 1

Regarding

1a) Considering per and post course surveys as separate studies, chances are they are highly pair-wise correlated, so they should not be considered independent samples. b) That seems likely; A type II error is incorrectly retaining a false null hypothesis (a "false negative").

2) If I use a non-parametric test, does that reduce the chance of type II error? It may, but that would not help if 1) is used.

3) Which test do you recommend I use? Is a paired test the right way to go, even though I will lose 5 data points? Paired testing does not assume independence, it is generally more powerful than non-paired testing. Likely more than enough increased power to overcome the loss of 5 unpaired test subjects, and with the information given, it is probably safest to do a non-parametric test, unless normal distribution conditions apply.

4) However, unpaired testing of the pre-course test data is suggested to see if there is a difference between the 5 student who did not complete the course, and the 10 who did.

Answer 2

The whole point of doing statistical analysis is to take data from a sample and make inferences about the population (or distribution) the sample was drawn from. Here you have information on the entire class (almost). What extrapolation or inference would you want a statistical test to help you with? Do you want to extrapolate (infer) to what would happen if there were more students, if you repeated the course...?

I suggest you just look at the data, and see what your students thought and how their opinions changed after the course. A statistical test will only be helpful if you want to make inferences about a larger population or distribution.