I am a teacher working on a simple research with the aim of studying whether the set of lesson designs I created can improve student performance. I have two groups with 15 students each. I divided them into the two groups making sure that each group has the same number of low, average, and high performing students. I had them take a pre- and a post-test. Unfortunately I am not that knowledgeable in Statistics. A friend told me to check for normality before even using t-tests to assess my students' performance. He's worried that the statistical power of the test may be too low because I only have 15 students in each group. How do you think should I proceed with the statistical tests?

Here are the pre-test scores of the two groups Group A- 10, 12, 14, 10, 11, 12, 13, 13, 24, 24, 20, 17, 27, 16, 29, 21 Group B- 10, 10,21,10,13,12,22,24,22,27,24,24,25,22,11,14 The highest possible score that a student could get is 50 and the lowest is 0.

  • $\begingroup$ None of these questions can be answered without seeing the data or summary statistics of it. $\endgroup$ Mar 1 '18 at 6:38
  • $\begingroup$ These are the summary statistics of their pre-test scores $\endgroup$ Mar 1 '18 at 7:48
  • $\begingroup$ Is it possible to include a histogram or a QQ-plot to the question? There's an edit button below the question. Also, what are the scores? Points given for questions? If so, what is the highest/lowest possible score? $\endgroup$ Mar 1 '18 at 8:30
  • $\begingroup$ Unfortunately, I cannot include any graph. I included the pre-test scores already, though. $\endgroup$ Mar 1 '18 at 9:42

In short, your data are not normally distributed, but as long as the groups are approximately normal, you can still perform comparisons with a simple $t$-test.

Checking the normality assumption

A $t$-test requires approximate normality of the groups. You can check to see if this is reasonable to assume using a normal quantile plot, or QQ-plot. Formally testing for normality is generally not a good idea.

If you are using $\textsf{R}$, I recommend the qqPlot() function in car, as it includes confidence intervals by default. Using the data in your example, you can see that there is no significant deviation from normality.

A <- c(10, 12, 14, 10, 11, 12, 13, 13, 24, 24, 20, 17, 27, 16, 29, 21)
B <- c(10, 10, 21, 10, 13, 12, 22, 24, 22, 27, 24, 24, 25, 22, 11, 14)



Note that this does not mean your data are normally distributed, they can't even be normally distributed, since they are integer values and the normal distribution is continuous. However, it does imply that a $t$-test is a valid way to compare these groups.

Choosing a $t$-test

An independent $t$-test requires that the observations in both groups come from distinct experimental units. This is the case when comparing A to B, but not when comparing before and after within a group. If you want to do this, you should use a paired $t$-test. For the independent $t$-test between A and B, you should also consider whether the groups' variances can be considered equal.

Lastly, if you intend to make more than one comparison, you should correct for multiple testing. A simple way to do this is by multiplying your $p$-values by the number of comparisons, known as the Bonferroni correction. Depending on what you intend to compare, this correction can easily drain power, so you could also try an FDR correction.

Alternative using regression analysis

While there are no deviations beyond the confidence intervals of the QQ-plot, notice how you can see large gap in the center of group B's QQ-plot (and more subtly in group A's), which is typical for binomial data. Since the scores are hidden ratios (number of points out of 50), a binomial regression model also makes more sense from a theoretical point of view. A more elegant way to model this would be with a binomial GLMM. In this context, you have repeated measures, which you can account for by estimating a random effect for student. In $\textsf{R}$, your model would then look something like this:

GLMM <- lmer(score/50 ~ group * time + (1 | student), family = "binomial")

This does come with a slightly harder interpretation of the results if you are unfamiliar with this type of model.

  • $\begingroup$ Thank you, Sir! I kind of get it now. I was wondering... I have a copy of the students' average grade in their Language class (the subject area concerned) and used it as a basis in dividing them into the control and experimental groups, making sure that the two groups have equal numbers of low, average, and high performing students. The Shapiro-Wilk Test showed that the two groups formed are "normal". I conducted an independent t-test concerning their average grades and since it was revealed that the mean of the grades of the two groups are not significantly different, $\endgroup$ Mar 2 '18 at 16:16
  • $\begingroup$ I concluded that the two groups are "comparable". Now I need to compare their mean pre-test scores, mean post-test scores, and their "gains" (post-test score minus pre-test score). Do I really have to test for normality for each set of data (pre-test scores, post-test scores, gains) before conducting paired and independent t-tests? Obviously, I have very poor knowledge in Statistics. $\endgroup$ Mar 2 '18 at 16:16
  • $\begingroup$ As explained in the linked answer, I explicitly advice against testing for normality. You should also be careful with your conclusions. If you don't reject the null-hypothesis, that does not mean you prove the null-hypothesis. $\endgroup$ Mar 3 '18 at 1:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.