What type of test and how many participants?

Question

I'm looking at collecting information on whether participants playing an educational game on a particular topic will achieve lower/comparable/superior gains between pretest/post-test scores compared to participants who learn that subject matter without a game (lecture only). Thus, I intend to randomly assign participants to one of two groups: game or no game.

Each participant will take a pretest and post-test.

Assume the results are as follows: all participants from both groups score an average of 50% on the pretests, the participants who play the game have an average post-test score of 90%, and the participants who don't play the game (lecture only) have an average post-test score of 70%. With the correct statistical technique and "enough" participants, it might suggest that the game made a difference.

Additionally, I also plan on examining whether gender affects pretest/post-test scores (thus, should I have four groups? Male/game, Male/no game, Female/game, Female/no game) I'll likely also look at other demographics like age (categories to be determined, but assume: age 20-40/game, 20-40/no game, 41-60/game, etc.)

Unfortunately, it's been over 20 years since I had a stats course. What technique do I use for these types of calculations (ANOVA?), and how many participants do I need in each category for the results to be valid? Additionally, what confidence interval is normally considered "valid"? 95%? 99%?

Answer 1

You are looking for "repeated measures AN(C)OVA". (The "repetition" part comes in because you will measure your participants multiple times, and someone who scores highly pre-test will probably also score highly post-test, which this model takes into account.) Another word for the same model is "linear mixed effects model".

In R, sample data and a first analysis could look like this:

library(nlme)
foo <- data.frame(score=runif(50),
    participant=rep(LETTERS[1:25],2),
    test=rep(c("pre","post"),each=25),
    age=rep(sample(20:30,25,replace=T),2),
    group=rep(sample(c("game","lecture"),25,replace=T),2))
model <- lme(score~group+age,random=~1|participant,data=foo)
summary(model)

As to how many participants you need (this is called "power analysis"), you will need to specify the expected variation around the means you supposed above, as well as the power you would like to have (i.e., how likely do you want to be to find an effect that actually exists). Given these inputs, you can simulate data and analyses as above for different sample sizes and see whether you find the effect you postulate often enough.

One other thing: you should not bin your ages. Leave them as they are, as numerical covariates. See, e.g., here.

Answer 2

Welcome to the site (and the field)! This sounds like a classic repeated measures ANOVA situation, to me. The groupings you described are known as factors, in ANOVA parlance. A part of running this test will be determining whether differences in the levels in each factor are associated with change in test score (e.g., Male v. Female, game v. no game, etc.). Importantly, before you can analyze whether the relationship between the levels in a specific factor are associated with significant change in your outcome variable, you'll first need to look for what are known as interactions in effects--that is, for example, whether the the relationship between gender and outcome differs depending on game v. no game.