Skip to main content
1 of 6
BruceET
  • 57.6k
  • 2
  • 36
  • 94

I don't know what assumptions you made in order to get your 'standardized effect sizes', so I will try to discuss this in a general way. The bottom line is that a paired design (having each subject take a test in both AM and PM) is almost always better than a two-sample design (testing different subjects in AM than in PM). So you should use the paired design if feasible.

Paired design:. Suppose I am one of your subjects in a paired experiment. The difference between my AM and PM scores will not necessarily represent the true difference between my ability to do statistics in the AM and PM. Probably I won't know all of the answers absolutely for sure on either the AM or PM exam. So I will do some guessing. Even so you could hope that the two tests are equivalent and independent measures of my ability on each occasion, so that if am substantially better at statistics in the PM, it is reasonable to expect that test results would give a rough idea how much better. [There are some cautions about how to conduct a paired experiment, which I discuss later, be let's ignore those for now.]

Two-sample design: Suppose I am one of two randomly chosen statisticians, and that you randomly assign me to the AM exam and the other subject to the PM exam. Maybe I get the lower score because I'm not very sharp in the AM, or maybe I get the lower score because the other subject (aside from any AM/PM difference s/he may have) is a better statistician than I am. [Again here, I am temporarily ignoring some cautions about how the experiment should be conducted.]

Comparison of paired and two-sample designs: Now suppose I am equally sharp in AM and PM, so in the paired design the only difference between AM and PM scores would be due to having to guess on a few questions. In the two-sample design, suppose neither subject has a true AM/PM difference. Due to possible differences in overall ability, the difference between our two scores is likely to be much larger than the difference between one statistician's variability due to guessing (and in addition to that, both of us probably do some guessing).

In any statistical test variability needs to be overcome by sample size in order to detect any real effect that may be present. Because the paired design has less variability, it will require fewer subjects in order to give useful results.

Issues for conducting the experiments:

Paired. (a) If you use exactly the same exam in AM and PM, then it is likely any subject will do better on the second administration, having had some time between to think about the problems or to look up some answers. So you need two equivalent and independent exams. Roughly speaking, questions should be of 'equally difficulty' and taking one of the exams should give no help for taking the second. (b) Even so, I think it would be best for a randomly-chosen half of the subjects to take the AM exam first and for the other half to take the PM exam first.

Two-sample. (a) A randomly chosen half of the subjects should be assigned to AM exams and the other half to PM exams. (b) Then you need to monitor that everyone show up. Otherwise, subjects randomized to AM, but who are less alert then, may self-select out of the study. Similarly, some subjects may self-select out of the PM study. To the degree that there is self-selection, the precautions of random assignment may be spoiled.

BruceET
  • 57.6k
  • 2
  • 36
  • 94