The scenario is as follows:

I want to test a new teaching method on non-native English speaking children to see if it is more effective that the current method. I therefore plan to have the following factors in my design:

A control group (existing teaching method); and An experimental group

Futher, I want to split these into students into those with a high standard of English vs those with low.

I will give them all a test at the start of term and then a similar one at the end of term. My null hypotheses are: 1) My new teaching method isn't significant 2) The English ability of the children prior to term isn't significant 3) There is no interaction between English ability and my new teaching method.

I'm fairly sure I need a factorial ANOVA model here, but cannot figure out whether it should be a 2x2 where I take the delta between the pre and post experiment test results as my 'values', or whether this should be a 2x2x2 design where the English ability, teaching method and time (before and after) are my factors.


1 Answer 1


I'll first provide an example, retaining the high vs. low distinction you made in your original question. Then, I will argue for why you should not distinguish between high and low—but use a continuous predictor instead.

I would recommend using a mixed-model (also known as multilevel or hierarchical models). You can think of your data being measured at two "levels":

  • Level 1: Measurement occasion
  • Level 2: Person

Your score is then predicted by measurement occasion (start vs. end), condition (experimental vs. control), and standard of English (high vs. low), as well as all of the two-way interactions, and the three-way interaction between the variables.

You can set up a model where occasion is a Level 1 predictor and the Level 2 predictors are condition and standard of English. The model would be:

Level 1:

$y_{ij} = \beta_{0j} + \beta_{1j}X_{ij} + e_{ij}$

Level 2:

$\beta_{0j} = \gamma_{00} + \gamma_{01}Z_j + \gamma_{02}W_j + \gamma_{03}Z_jW_j + u_{0j}$

$\beta_{1j} = \gamma_{10} + \gamma_{11}Z_j + \gamma_{12}W_j + \gamma_{13}Z_jW_j + u_{1j}$

Where $i$ is the subscript for Level 1 observations and $j$ is the subscript for Level 2 observations, $y$ is the score on the test, $X$ is the measurement occasion, $Z$ is the treatment condition, and $W$ is the standard of English.

Written in the syntax of the popular lme4 R package, this formula would be:

score ~ time * condition * standard_of_english + (1 + condition * standard_of_english | id)

Where id would be an identification variable for each of your participants.

So why use this instead of an ANOVA? Two reasons:

First, the 2x2 fully between-subjects analysis relies on difference scores. Using difference scores is something hotly contested. A classic reference on this is Edwards, J. E. (2001). Ten Difference Score Myths. Organizational Research Methods.

Simply avoid all that hassle and problems with difference scores by not using them! So then that leaves you with being able to use the 2x2x2 mixed-analysis ANOVA, where the within-subjects factor is measurement occasion (before vs. after). The problem here is that you have to dichotomize something that is fundamentally continuous. You are losing a lot of information in doing this, and splitting a continuous variable into two groups is pretty roundly panned these days. A classic reference for this is MacCallum et al. (2002). On the Practice of Dichotomization of Quantitative Variables. Psycholgical Methods..

What is great about using the mixed-model approach I described above is that $W$ (or standard_of_english) in the model (or lme4 formula) above can be continuous—you do not have to cut it into two groups and lose lots of information.

So, my suggestion would be to use the mixed-model, described above.

  • $\begingroup$ Thanks for the great, in-depth reply Mark. I'll take a look at your suggestion $\endgroup$
    – Iain
    Nov 3, 2017 at 0:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.