7
$\begingroup$

I am trying to evaluate a new curriculum. I intend to do it by giving students a test at the beginning of the academic year (pretest) and one at the end of the year (post test).

There are roughly 50,000 students and so I can only do the evaluation on a sample size. For this type of evaluation that compares the pretest and post test scores, how do I calculate sample size?

I am a novice in statistics and research and would greatly appreciate any help with this.

$\endgroup$
1
  • $\begingroup$ Have pre/post-tests been done in previous academic years with the old curriculum? And is the new curriculum going to happen anyway - and will all 50,000 students be doing it? If running old and new curriculum in parallel is not an option, the two answers so far are not so useful. Do you already have in mind what a good evaluation will look like? $\endgroup$ Commented May 24 at 7:31

2 Answers 2

6
$\begingroup$

Causality and Testing

I don't know if this a valid way of testing this. What you are essentially doing is setting up a test to see if there are differences between their initial scores and their final scores. But this doesn't tell you at all if this can be attributed to your curriculum. You may just have a bad class, you may have an odd distribution of students (e.g. a high number of med students in a psychology class), or it may be even teacher related (perhaps your enthusiasm about the curriculum is projected in the class and biases the effects in some way). In fact, the most likely outcome is simple growth in the outcome...they will just get better as the semester progresses simply by having attended class, regardless of whether or not they get the new curriculum.

The only way to really make any causal associations is to have some kind of counterfactual you can compare against that says basically "if my students hadn't been given the test, what would have happened? And that's where you need a control group. Perhaps you can teach two different sections of the class (Section A and Section B), where one is tested with the curriculum while the other doesn't get the curriculum. This presents some other potential contaminants (e.g. the earlier class is less alert and may perform worse just on that point alone). So you could try to control other elements, such as having two teachers teach the same class under highly standardized instructions. The key is to find a condition where there are as few sources of contamination of the effect as possible, so that you can justifiably attribute it to the intervention (the curriculum).

As far as what test can achieve this, a simple t-test may be all you need to compare the groups. You may consider adding some sophistication of the design if you think there are other confounds (age, gender, etc.), for which perhaps a regression would serve a better purpose. That will depend entirely on which variables you deem necessary for investigating this question.

Sample Size

As far as sample size, this can be easily achieved by conducting a power analysis. Often people use GPower for it's ease of use in designs as simple as yours. For something more complicated, you can turn to simulated power analysis, but given you noted your lack of statistical prowess, that can be ignored for your purposes, particularly given the ease of which this can be analyzed.

$\endgroup$
6
$\begingroup$

Shawn gave an excellent answer. I'm just adding to it.

I am glad that you write you "intend to do it". That implies you haven't done it yet. You should change your design in precisely the way Shawn suggested: Get a control group. You really should take a course in experimental design (or, if this is a rush job, hire someone to help you), but see below for one idea. If you do the test just on one group, all of whom take the curriculum, then there is no result which will let you say anything interesting.

for a design, I suggest getting a control group that takes the old curriculum. Then, test multiple times during the year. Then use a multilevel model and evaluate time, curriculum, and the interaction of the two (which is what you are really interested in) and also add useful covariates. Even in a multilevel model, covariates can be useful.

Power analysis for such a model is complex.

So, again, I suggest hiring someone to do this for you, or, if that's not possible, doing something simpler.

For power analysis more generally, you need to know all but one of the following:

  • Desired power (often 0.8 or 0.9, but can be different).
  • Tolerable type 1 error (usually 0.01 or 0.05, but can be different)
  • Type of test (t, or multiple regression or whatever)
  • Sample size

then, if there is a program already (e.g. PASS or GPower or power calculations in R, etc.) you can just plug things in but, as Shawn said, if you do a complex analysis, you many need to simulate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.