Dependent (Paired) T-Test Sample Variance

Question

When conducting a repeated measures test where one measures the before/after values for several pairs, how does one account for variance within the pairs themselves? For example lets look at this example of 3 student scores SAT math section as the result of a prep-course.

At this level, it seems that the test was significant in improving their overall score. However, what if I now add in the information that each student has taken the exam 3 times prior to the class and their scores varied +/- by 50 points each time. Now the 30 point increases no appear to be relevant.

How does one account for these types of scenarios when running paired t-tests?

Answer 1

I'm not an expert, but I happened to come across The Handbook of Biological Statistics searching on a related question right before seeing your question. It suggests a two-way ANOVA with replication, but it seems like it's not so simple.

I'll try to map your question to the language of the Handbook and walk through the analysis. Your two "ways" (factors, nominal variables) would be the identity of the student, and pre- vs. post-course. The two-way ANOVA tests 1) whether the means differ by student, 2) whether the means differ pre- and post-course, and 3) whether the identity of the student and pre- vs. post factors affect each other (the "interaction test"). In a repeated-measures design like yours, you don't care about 1 (of course students' abilities differ), you do care about 2, and 3 tells you whether 2 is meaningful.

So, assuming you find a significant difference pre- vs. post-course, there are two outcomes from the interaction test: if the interaction is not significant, you can say, "the course improves scores." If it is significant, you have to add, "...for some students but not others."

Your case, where you have three measurements before the treatment (course) and only one after, looks to be an unbalanced design. According to the Handbook:

When the sample sizes for the subgroups are not equal (an "unbalanced design"), the analysis is much more complicated...

There are pointers to software in the article; hopefully you can find a package that handles unbalanced designs.