I calculated the two standardized effect sizes (i.e., once for paired samples and once for independent samples) and got two different values (0.45 and 2.58). Why are they different?
1
-
1$\begingroup$ "once in the morning and once in the evening" -- if you did it in only that order you can't compare the scores, because evening vs morning would be confounded with the practice effect. $\endgroup$– Glen_bCommented Oct 28, 2018 at 1:38
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I don't know what assumptions you made in order to get your 'standardized effect sizes', so I will try to discuss this in a general way. The bottom line is that a paired design (having each subject take a test in both AM and PM) is almost always better than a two-sample design (testing different subjects in AM than in PM). So you should use the paired design if feasible.
Paired design:. Suppose I am one of your subjects in a paired experiment. The difference between my AM and PM scores will not necessarily represent the true difference between my ability to do statistics in the AM and PM. Probably I won't know all of the answers absolutely for sure on either the AM or PM exam. So I will do some guessing. Even so you could hope that the two tests are equivalent and independent measures of my ability on each occasion, so that if am substantially better at statistics in the PM, it is reasonable to expect that test results would give a rough idea how much better. [There are some cautions about how to conduct a paired experiment, which I discuss later, be let's ignore those for now.]
In a paired design each subject is a mini-experiment in his or her own right. The difference between AM and PM performance for each subject tells you something.
Two-sample design: Suppose I am one of two randomly chosen statisticians, and that you randomly assign me to the AM exam and the other subject to the PM exam. Maybe I get the lower score because I'm not very sharp in the AM, or maybe I get the lower score because the other subject (aside from any AM/PM difference s/he may have) is a better statistician than I am. [Again here, I am temporarily ignoring some cautions about how the experiment should be conducted.]
In a paired experiment each AM subject can be compared only with the overall average of all the PM subjects. (This is a much less-direct comparison than a single subject's AM/PM comparison in a paired design.)
Comparison of paired and two-sample designs: Now suppose I am equally sharp in AM and PM, so in the paired design the only difference between AM and PM scores would be due to having to guess on a few questions. In the two-sample design, suppose neither subject has a true AM/PM difference. Due to possible differences in overall ability, the difference between our two scores is likely to be much larger than the difference between one statistician's variability due to guessing (and in addition to that, both of us probably do some guessing).
In any statistical test, variability needs to be overcome by sample size in order to detect any real effect that may be present. To the extent that a paired design has less variability than a corresponding two-sample experiment, it will require fewer subjects in order to give useful results.
In most cases, if a paired experiment requires $n$ pairs ($2n$ exam administrations), in order to have the same ability to detect an AM/PM difference in ability, a two-sample experiment will require many more than $n$ people in each of the two groups. So for all the difficulties in getting subjects to show up twice for testing, a paired experiment may really be easier to conduct than an equally-powerful two-sample experiment.
Generally, experienced experimenters always use paired experiments in preference to two-sample experiments, provided that a paired experiment is feasible. In drug testing, it is often possible to do paired experiments for pain relievers because the effect of a pain-relieving drug is temporary and second (independent) administration of a different drug is feasible. By contrast, a trial comparing drugs to slow the progress of a cancer often has to be a two-sample experiment.
Issues for conducting the experiments:
Paired. (a) If you use exactly the same exam in AM and PM, then it is likely any subject will do better on the second administration, having had some time between to think about the problems or to look up some answers. So you need two equivalent and independent versions of the exam. Roughly speaking, questions should be of 'equally difficulty' and taking one of the exams should give no help for taking the second. (b) It would be best for a randomly-chosen half of the subjects to take the AM exam first and for the other half to take the PM exam first. Also, it would be best for half of the AM exams to be Version 1 and half Version 2; with everyone taking a different version in AM than in PM.
Two-sample. (a) A randomly chosen half of the subjects should be assigned to AM exams and the other half to PM exams. (b) Then you need to monitor that everyone shows up. Otherwise, subjects randomized to AM, but who are less alert then, may self-select out of the study. Similarly, some subjects may self-select out of the PM study. To the degree that there is self-selection, the precautions of random assignment may be spoiled.
