Critical effect sizes and power for paired t test

Question

Is there some sort of easy way to calculate the required sample size for a paired t-test with an effect size of 2 standard deviations? In my line of work, we often use an effect size of 2SD from the mean for power and sample size calculation with independent t-tests but I'm not sure how to adapt that to a paired t-test. I'm not even sure if that typed of effect size makes sense for a dependent t-test. Thanks for the help.

Answer 1

Yes, this is possible and even fairly easy, but additional information is required. Specifically, we have to make an assumption about what the correlation between the observations from each pair are.

The effect size as a difference in standard deviation units is usually referred to as $d$. We can apply a correction factor to $d$ to incorporate the information about the aforementioned correlation, and then we can use our standard power formulae with this corrected $d$ (making sure to also mind the change in degrees of freedom associated with moving to the paired design) to compute power. The corrected $d$ is $$ d_o = \frac{d}{\sqrt{1-r}}, $$ where $r$ is the correlation. I have called this $d_o$ because this is sometimes referred to as the "operative effect size."

Here is a little R routine that computes a table of minimum number of PAIRS as a function of the assumed correlation and the desired power level, with $d=2$ assumed.

library(pwr) # package for pwr.t.test() function
             # may need to install first with install.packages()

# define a function to get the minimum number of pairs
# for a given correlation and desired power level
getN <- function(r,p){
  unlist(mapply(pwr.t.test, d=2/sqrt(1-r), power=p,
    MoreArgs=list(n=NULL, sig.level=.05, type="paired"))["n",])
}

# apply this function to all combinations of the parameters below
tab <- outer(seq(0,.95,.05), c(.7,.8,.9,.95,.99,.999), "getN")
dimnames(tab) <- list("Correlation"=seq(0,.95,.05),
                  "DesiredPower"=c(.7,.8,.9,.95,.99,.999))
tab

Which returns the following:

           DesiredPower
Correlation      0.7      0.8      0.9     0.95     0.99    0.999
       0    3.767546 4.220731 4.912411 5.544223 6.888820 8.656788
       0.05 3.691858 4.126240 4.787326 5.389850 6.669683 8.350091
       0.1  3.615930 4.031562 4.662220 5.235637 6.451021 8.044096
       0.15 3.539645 3.936653 4.537050 5.081483 6.232774 7.738792
       0.2  3.462940 3.841433 4.411750 4.927417 6.014903 7.434270
       0.25 3.385708 3.745774 4.286234 4.773338 5.797404 7.130529
       0.3  3.307922 3.649640 4.160447 4.619143 5.580267 6.827580
       0.35 3.229382 3.552889 4.034209 4.464751 5.363362 6.525430
       0.4  3.149970 3.455310 3.907393 4.309986 5.146613 6.224026
       0.45 3.069435 3.356743 3.779777 4.154653 4.929824 5.923282
       0.5  2.987581 3.256903 3.651065 3.998456 4.712773 5.623032
       0.55 2.904079 3.155423 3.520841 3.841020 4.495111 5.323066
       0.6  2.818472 3.051834 3.388672 3.681805 4.276260 5.022875
       0.65 2.730145 2.945449 3.253781 3.520048 4.055501 4.721751
       0.7  2.638237 2.835369 3.115118 3.354639 3.831565 4.418560
       0.75 2.541442 2.720152 2.971074 3.183823 3.602697 4.111397
       0.8  2.437713 2.597460 2.819127 3.004879 3.365682 3.796879
       0.85 2.323340 2.463226 2.654597 2.812710 3.114890 3.468745
       0.9  2.190677 2.309002 2.467897 2.596901 2.838233 3.113596
       0.95 2.018024 2.110699 2.231866 2.327720 2.501567 2.692358

Note that $d=2$ is considered in many fields quite a large effect size, so the resulting minimum numbers of pairs are all quite low.

Answer 2

Any effect size is possible, it's hard to tell what you mean by "makes sense".

There are many power calculators on the internet found with a simple Google search. You'll see that G*Power is often recommended here.

Conceptually you calculate power more or less the same way. In the independent case you're probably using the pooled variance of your conditions to get an SD. That's a Cohen's D. However, with the repeated measures case it will be the SD of the effect, which incorporates the covariance between the conditions. If you have a well justified repeated measures design the covariance should be fairly large, which will result in a small effect SD. Therefore, you'll require a smaller N. You need to somehow find this effect SD or covariance (or correlation). If there are no repeated measures studies from which to derive it or you don't have prior data you might better off taking a different approach (at least for your first repeated measures study).

You don't have to calculate power before conducting a study. You could take the following approach. You could decide how much of an effect you care about. Let's say it's 4 (not 4SDs, 4 on your scale, whatever it is...it could even be your original 2 SDs, although that's rather large). Effects smaller than that wouldn't really mean much even if the test was significant and you want to be able to detect effects at least that size. Run subjects until you can detect an effect of that size (calculate the width of an effect confidence interval). Stop once you can detect the desired value. I've previously posted the simulations that show that this does not inflate alpha appreciably, and gives you the expected power and N on average. (Keep in mind that you are not allowed to make decisions on significance of a test, only sensitivity. That requires discipline.)