Paired t-test to support increase in student reading scores from fall to spring

Question

I did a pre-test, post-test method and I tried to perform a t-test to show how reading scores increased from fall to spring.

I’m not sure if my numbers are accurate.

What throws me off is the NULL Hypothesis. According to my calculation using Excel it is $7.37 \times 10^{-27}$ or 0.00000000000000000000000000737 (for a paired, 2 tail t-test). Is this realistic to get? My understanding is that you have to achieve <.05 to reject the null hypothesis. And my calculated t-tests are really small. Does this mean I can reject the null hypothesis and say that what I did to increase reading scores was effective?

Also, some students who took a test in fall withdrew from my class mid-year and didn’t take a spring test. Do I leave those cells blank or do I fill them in with zeros? Or do I just pretend that those kids never existed? How much would that throw off the results? Thanks.

Answer 1

What throws me off is the NULL Hypothesis. According to my calculation using EXCEL its 7.37481E-27 or 0.00000000000000000000000000737 (for a paired, 2 tail TTEST). Is this realistic to get?

It depends on the number of students and difference between pre and post. I think there is probably a problem with your calculation however.

My understanding is that you have to achieve <.05 to reject the null hypothesis. And my calculated TTESTS are really small.

The number .05 is arbitrary and based on an example Ronald Fisher used ~80 years ago. He later said that no one should use the same number for every case, it depends on the circumstances. Your p-value is very small so I am sure would be considered "significant" by anyone, however there is probably a problem with your calculation.

Does this mean I can reject the null hypothesis and say that what I did to increase reading scores was effective?

Rejecting the null hypothesis only means that the pre and post scores were not exactly the same. It could also be significant if the students were worse in later semester. Or the scores could be different for some other reason besides your teaching (a popular tv show ended so they studied more, it could be anything).

It is best to plot your results to show the effect. Plot a line for each student between pre and post. Did some students improve while others did not, or even got worse? Perhaps there is something in common regarding who got better and who did not (if this is the case) that could help you improve your teaching strategy even further. Most likely you do not only care about the "average student" but more how to help each individual student.

Also, some students who took a test in fall withdrew from my class mid-year and didn’t take a spring test. Do I leave those cells blank or do I fill them in with zeros? Or do I just pretend that those kids never existed? How much would that throw off the results? Thanks.

Do not fill them in with zeros. It would probably be easiest for you to drop them from the significance test analysis (pretend they didnt exist). But you should not ignore the data! For example, if there was a large difference pre and post for all the other students and the ones who dropped the course were the ones with high scores the first semester this may skew your results. Perhaps they withdrew because of the teaching method, etc. There is no statistical test for this, you just have to think about how to explain what happened to generate the data you got.

If you post the calculation you did I can help further, still I think it is more important to plot the data and look at the effect for each student rather than calculate a p-value.

Answer 2

(1) P-value: How different are the two values and how large is the sample? If these two values are large the p-value sounds reasonable. Usually a p-value like this is just given as <0.001. Using more decimal places doesn't really make sense to me. P=0.0001 isn't very different in my mind from p=0.000001. Smaller isn't more meaningful.

(2) This is sort of what I would expect. Depending on age of students I would be surprised if the fall/spring values were the same. (If these are college students this is an impressive difference, but if 6th graders makes sense)

(3) Missing values is a difficult issue. Statisticians build careers on this sort of stuff. In your case I'd just use "casewise deletion" (also known as "listwise deletion") where you only include the cases where you have both pre and post results. If you want to go the extra step you can compare the students for which you have full information to those without full information to see if they are different in some way. The concern is that your results might be biases since your excluding students that may be different from the ones not excluded. (technical jargon for this would be that the data might not be MCAR, but MAR or NMAR)

There are methods to better adjust for missing data, but the ones I can think of require a lot of statistics and I think are unlikely to add much to the analysis. Others on this site may have better ideas on this issue.

Answer 3

There are several types of t-tests that use different formulae, although they are based on the same idea. So you would need to make sure you are using the right one— single sample, matched/paired, independent samples with same sizes, or independent samples with different sample sizes. A typical pre-test/post-test would be matched/paired, although this may not fit in your case. First, that usually presumes you have the exact same subjects, the same number of subjects, and the same test. If you aren’t using the same assessment measure, then it would hard to call this pre-test/post-test, even if it is the same student group, and having students withdraw makes it even more difficult to use paired/matched. You would have to completely remove from the analysis the students that dropped out of the class. An independent samples test would be the other option in this case, and since students have dropped out, you have different sample sizes. Part of this decision is whether or not you are using the same assessment before and after, or if you are simply comparing grades—in the latter case, it’s not pre-test/post-test.

Regarding your null & p-value, I’m not sure why you interpret that as problematic. Regardless of which t-test you choose, you are trying to see if there is a difference between the mean scores of the two assessments. Your null is “there is no difference between the mean scores,” and your alternate is “there is a difference between the mean scores.” If you are using alpha=0.05, then you are looking for p<0.05 to reject the null, which means that you can be 95% confident of not committing a Type 1 error when you say there is a difference between the mean scores from fall to spring. The number you got is just really small, but regardless, it’s less than 0.05. Sometimes you can get a really small p-value if you have really large sample sizes—even a small difference in the means can show up as significant. What you might be more interested in is to look at the effect size, to see if the difference is meaningful. The significance test (t-test & p-value) only indicates whether you can say the means are different, not whether that difference is important.

If you are not confident that Excel did what you think it did, there are several online tools that allow you to enter data to calculate t-values, for example: https://www.graphpad.com/quickcalcs/ttest1.cfm

You can also find online calculators for the effect size, for example: http://www.socscistatistics.com/effectsize/Default3.aspx or https://www.uccs.edu/~lbecker/