I've just been given an old set of lecture notes to look at.

One of them contains the following question: if we are undertaking a paired $t$-test with a mean of differences $d=1$ and standard deviation $s=2$, what is the minimum number of subjects that would be required to "ensure that the difference observed is significant". Let's leave aside for now that particularly monstrous phrasing.

The "model answer" is as follows: the test statistic we wish to discuss is $\frac{d}{s/\sqrt{n}}$. Let's find a value of $n$ such that this test statistic is greater than the corresponding critical $t$-value. It transpires that this is about 19:

deeff<-18; c(qt(p=1-0.05/2, df=deeff), 1/(2/sqrt(deeff)))
[1] 2.100922 2.121320
deeff<-19; c(qt(p=1-0.05/2, df=deeff), 1/(2/sqrt(deeff))) 
[1] 2.093024 2.179449

Now, clearly if we do a power test "properly", using the above as a definition of Cohen's d, we get a very different estimate of n:

pwr.t.test(d=0.5, type="paired", power=0.8)
>   Paired t test power calculation 

              n = 33.36713
              d = 0.5
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

NOTE: n is number of *pairs*

Now, clearly, this is the result of a huge conceptual misunderstanding. My question is this, what's the most concise way to explain why the original answer is wrong?

  • $\begingroup$ How do you explain the output from your calculation of power using qt? $\endgroup$ – AdamO Feb 5 '18 at 17:27
  • $\begingroup$ "Ensuring the difference observed is [statistically] significant" means actually stating the desired power of the test. There's nothing wrong with doing power calculations. But nowhere is it said what kind of power you are trying to get. The two approaches are calculating sample sizes for different powers. Of course, for a power of 1, you need everybody. $\endgroup$ – AdamO Feb 5 '18 at 17:39
  • $\begingroup$ @AdamO -- the argument may be that you'd need to get $n\in[18,19]$ with sample $s$ and $d$ equal to the population mean. It's an incorrect argument, but it's what the original lecturer made and I can imagine that some students may find it attractive. I'm very much in favour of doing power calculations properly; I was just looking for help coming up with a pithy, understandable explanation for exactly why this is wrong. $\endgroup$ – Landak Feb 5 '18 at 19:19
  • $\begingroup$ An argument to what? And how can a sample be equal to a population mean? Just to clarify, are you saying that the R code shown in the first set of displays is the power calculation the professor used? $\endgroup$ – AdamO Feb 5 '18 at 19:30
  • $\begingroup$ Where does a power specification of 80% in the second calculation come from? That doesn't seem to have anything to do with the question. $\endgroup$ – whuber Feb 5 '18 at 20:52

I assume that the d = 1 and s = 2 values are descriptions of the population from which the sample is drawn. If that's the case, then any sample you draw may overestimate or underestimate the values for d and s. However, you want to ensure (horrible phrasing aside) that the difference is significant even if your sample underestimates d or overestimates s. This is why your second power calculation results in a larger sample. N = 19 may produce a significant effect if d = 1 and s = 2 in the sample you draw, but N = 33-34 will result in a significant effect for 80% of the samples you could draw, given the population values provided.


I calculate the power as the following:

The critical value when the null is true: cv <- qt(p=1-0.05/2, df=33, ncp=0, lower.tail=T)

The probability of exceeding the critical value when the alternative is true:

pow <- pt(q=cv, df=33, ncp=1/(2/sqrt(33)), lower.tail=F)

which is 80% as the power calculator mentions. I think the first calculation is wrong. Whether that's the original or not doesn't matter I guess.

The reason why the original is wrong is that the expected test statistic exceeds the critical value when the power is usually only about 50% because the non-central T-distribution is mostly symmetric. Running those numbers actually verifies this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.