I am running an experiment on students in two of my classes. There is an online class vendor that promises higher test scores with only 2-3 hours more work per week. I decided to give one of my classes access (test) and the other class no access (control).

Both classes are above 40 people and the difference between the mean is ~3.6 points (with the online class with higher score). After I ran a z-test on the population I've found that the difference in mean is significant using stats.ttest_ind.

With this information in mind - can I say that the online class provided my students with 3.6 points in test scores? Or does the z test only say there is a difference in means but it may not be 3.6 points?

Thanks in advance!


First you should look into the assumptions behind the test that you are using.

Generally the z-test is used when you know the population standard deviation, use the t-test when using sample standard deviations to estimate the population SD. However, with both samples larger than 40, the difference in results will be small.

Both the z-test and t-test are based on the assumption of normality which can be satisfied if either you know that the true distribution is normal, or if the sample size is large enough and the data is normal enough. You should check the observed scores for outliers or strong skewness before using either test.

The big issue with how you described your data collection is independence of the data. Scores from students within the same class are not likely to be independent of each other. For example, a student in one class may have asked a question that you then answered (and everyone in the class heard the answer), but the other class did not hear the answer, only heard your initial description in the lecture. Since this affects everyone in one class (or everyone who attended that day) and none in the other class, that breaks the independence requirement. The better design would be to have multiple classes receive each treatment so that you can assess the amount of variability between classes within each treatment and take the classes (clusters) into account in the analysis. As it is, without a very strong assumption about independence, you really have 0 degrees of freedom for the test and cannot conclude much.

Even when all the assumptions and conditions are met (or at least reasonable), you need to be careful about stating a cause and effect relationship. Using a word like "provided" implies a causal relationship, which requires stronger conditions than just saying that a difference was observed.

You should also look at the confidence interval, this gives more information and accounts for more of the variability than just the point estimate.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.