2
$\begingroup$

What would be the most appropriate statistic to use if one wanted to see whether an instructional treatment produced learning gains in a classroom in which all students were tested before and after the treatment? Assume the learning measures are valid, but the pre-test has, say, a score that can range from 0-10 but the post-test can go from 0-100 (imagine a pre-test that's a short quiz and a post-test that's a longer test, for example).

I recognize the subjects represent matched pairs. If various assumptions are met, it seems like the null hypothesis could be tested with a matched-pairs t-test? Is that right? If so, what scores (numbers) are used in the actual test, given how different the two scales are? Do you have to do something like transform each raw score into a percentage?

Improve this question
$\endgroup$
4
  • $\begingroup$ It's not clear from your problem statement that there is a control group. Is this the case? If you only have a treatment group, then you cannot estimate the treatment effect because it is confounded with the fact that students learn when they go to school. $\endgroup$
    – dipetkov
    Apr 11, 2022 at 7:06
  • $\begingroup$ @dipetkov No, no control group. It's a hypothetical (though common?) situation I presented to illustrate the question, rather than an actual study. As an actual study, your critique is spot on. $\endgroup$
    – Al C
    Apr 11, 2022 at 17:29
  • $\begingroup$ If the pre-test and the post-test measurements are different, I don't see how they can be compared. If you want to have a proper pre-test/post-test comparison, you have to run these tests so that they give comparable results. $\endgroup$
    – Christian Hennig
    Apr 11, 2022 at 20:44
  • $\begingroup$ Are the pretest questions a subset of the posttest questions? Or are they completely different? $\endgroup$
    – Noah
    Apr 12, 2022 at 14:53

1 Answer 1

Reset to default
1
$\begingroup$

You cannot determine whether a treatment has an effect without a comparison to a control group and you cannot use the pre-treatment scores as the control, at least not convincingly.

Let's elaborate on your hypothetical situation: You give students a test at the start of the semester, give them the treatment, then test them again at the end of the semester. You cannot use the pre-treatment score as the control for the post-treatment score because students will learn something during the semester with or without the treatment. In other words, the effect of the treatment is confounded with the fact that students learn when they go to school.

Since this is a hypothetical experiment, however, suppose that you have pre and post measurements for a control group as well (eg. another class in the same school).

You can model the data you've collected with a regression: the predictors are the pre-test (X), an indicator variable for whether a student received the treatment or not (T) and any covariates (Z); the outcome is the post-test (Y). The effect of the treatment is the coefficient $\beta_T$ of the treatment indicator and you can test whether $\beta_T$ is greater than zero.

Regression it doesn't assume that the slope of Y on X is 1; instead it estimates it. This means that regression elegantly handles the case when pre and post scores are on a different scale, unlike gain scores (gain = post score - pre score) which implicitly assume that the slope is 1.

You can learn more about how to estimate a treatment effect in Chapter 19 of Regression and Other Stories [1]. The textbook uses an educational study as a running example and discusses many subtleties of performing and analyzing such an experiment (eg. one treatment and one control classroom won't be enough because the treatment effect might be confounded with a teacher effect.)

[1] A. Gelman, J. Hill, and A. Vehtari. Regression and Other Stories. Cambridge University Press, 2020.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I understand your point. Your study design would be much better than mine. What I'm trying to find out, though, is less the study design and more simply the test to use when data from the samples being compared are scored differently. To follow up on your thoughtful response, you wrote "students will learn something during the semester with or without the treatment." Suppose, hypothetically, that's what was being tested. And the measures at the beginning and end of the semester assessed similar knowledge, with scales that were really different--one was 0-10 pts and the other 0-100 pts. $\endgroup$
    – Al C
    Apr 11, 2022 at 22:01
  • $\begingroup$ A short quiz and a long test might be both valid measures of educational attainment and not be comparable at the same time, ie., 3 on the 0-10 quiz scale might not correspond to 30 on the 0-100 test scale. $\endgroup$
    – dipetkov
    Apr 11, 2022 at 22:31
  • $\begingroup$ With the quiz, being shorter, it's also more likely to run into a ceiling effect. So not only do you need to administer the same scale, you have to choose it carefully so that it's possible to detect improvement. $\endgroup$
    – dipetkov
    Apr 11, 2022 at 22:47
  • 1
    $\begingroup$ (+1) And the study design you recommend is typically called a BACI (before-after-control-impact) design. $\endgroup$
    – mkt
    Aug 21, 2022 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.