I am performing a significance test on the gain score between a Pre-Test (say Y1) and a Post-Test (say Y2) for a Treatment Group and a Control Group.

Let me give a brief background on the experiment. My client is working with a large group of schools on improving the learning levels of kids in Math. Out of this large group 653 students were tested in the Pre-Test. There was also a control group which had 213 students who were also tested. After this the intervention group received special coaching (intervention) along with regular classes. And the control group just had regular classes. About 6 months later, a Post-Test was conducted. But this time 739 students (different set of students from the intervention group, may have some overlap) from the intervention group and 210 students (pretty much the same group as the Pre-Test) from the control group were tested.

The problem I am facing is how to do a significance test on the difference between the Gain Scores. Here is the method which I am planning to employ.

Since I am sampling large enough samples from the Intervention Group and the Control Group for the Pre-Test, and I have point estimates for the Mean and SD, the CLT gives me a normal distribution for the Pre-Test for both Intervention and Control. The same argument applies to the Post-Test. Now I assume that the Pre and Post Test performances are independent. Now I can combine Pre and Post test normals to create a new normal distribution for Y2-Y1. The same argument again applies to control also. Now I want to apply Welch's t-test to check for statistical significance between the groups. But I don't have the degrees of freedom and number of observations for both Intervention and Control. I'll use the Welch-Satterthwaite Equation to get the Degrees of freedom for each group and then add one to get the number of observations. After this I can apply Welch's t-Test.

When I am doing this I get extremely high t-statistic values, So I am not really sure whether I am doing it the right way. Could someone please tell me if there is a flaw in my logic.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.