2
$\begingroup$

For a research I want to do a t-test between two populations. Since I have a fairly small sample size I want to calculate the necessary sample-size for getting a good power (0.8 is supposed to be very good).

I did calculation as below in R using the pwr.t.test() form the package "pwr". It uses Cohen's d, sample size, significance level and power, i.e. three of these to determine the fourth.

I chose a significance level of 0.05, a random power of 0.5 and then calculated Cohen's d which had me a value -0.01183773. I'm not even sure, how to handle a negative Cohen's d. I've read can be negative.

m1<- 0.3133333  ## population 1
m2<- 0.6766667   
m3<-  -0.4866667
m4<- 0.6566667
m5<- -0.04666667  ## population 2
m6<- 0.8066667
m7<- 0.54
m8<- -0.1066667

pooledSD<-sqrt((sd(c(m1,m2,m3,m4))+sd(c(m5,m6,m7,m8)))/2)
C<-(mean(c(m1,m2,m3,m4))-
  mean(c(m5,m6,m7,m8)))/
  (pooledSD)

In the end-effect I obtained a sample size of 54821.11 using the function an the Cohen's d obtained.

pwr.t.test(d=C, sig.level = 0.05, power = 0.5, 
           type = "two.sample",
           alternative = "two.sided")

Typing into sample size n = 4, leaving out power, gives me a power of 0.05

pwr.t.test(n=4, sig.level = 0.05, d = C, 
           type = "two.sample",
           alternative = "two.sided")

I have four samples in a bad scenario.

Question 1: Did I calculate Cohen's d correctly? Question 2: Does power of 0.05 using this function mean that power is non-existent?

Thanks!

$\endgroup$
  • $\begingroup$ For a start, double check your formula for pooledSD. $\endgroup$ – EdM Apr 11 at 13:55
4
$\begingroup$

Let's plot the data points you use.

data

plot(c(rep(1,4),rep(2,4)),c(m1,m2,m3,m4,m5,m6,m7,m8),pch=19,xlim=c(.5,2.5),
  xlab="",xaxt="n",ylab="",las=1)
lines(c(.8,1.2),rep(mean(c(m1,m2,m3,m4)),2))
lines(c(1.8,2.2),rep(mean(c(m5,m6,m7,m8)),2))

The horizontal lines give the group means.

  1. Your calculation is almost correct.

    Your calculation of the pooled standard deviation is slightly wrong. You need to square the groupwise SDs to get groupwise variances, then take the average of those and finally take the square root. You forgot to take the square. Here is how it should look like:

    pooledSD <- sqrt((sd(c(m1,m2,m3,m4))^2+sd(c(m5,m6,m7,m8))^2)/2)
    

    However, that doesn't make all that much of a difference. The correct calculation yields a Cohen's $d$ of $d=-0.01673617$.

    Recall what Cohen's $d$ is: it is the difference between the group means, standardized by the pooled standard deviation. In your case, this means that the group means differ by only 0.01673617 pooled standard deviations. So, essentially not at all. See the picture above, especially the horizontal lines (the group means).

    A negative value for Cohen's $d$ is fine. It just depends on how you take the difference in group means. Unless you have a one-sided hypothesis, this doesn't make a difference.

  2. A large sample size in your original calculation, or a tiny power using your sample size of $n=4$ per group, are two aspects of the same problem: you are trying to detect an infinitesimally small effect, namely a difference in group means that is no more than 0.01673617 pooled standard deviations. To detect such a small effect with power 0.5, you need a huge sample size. Or conversely, if you have a small sample size, you have essentially no chance of detecting it. (Note that the power will not drop below your alpha level. Which just means that if you do get a significant result, it's more likely to be a false positive.)

Bottom line: it's good that you are thinking about your power before starting your study. Unfortunately, if your effects are truly this minuscule, you will not be able to detect them with a small sample.

$\endgroup$
  • $\begingroup$ Thank you! I do understand. The thing that I don't understand is that once you'd have means that are far apart and that gap is similar to the SD's you get high statistical power. Nevertheless, you still have the same sample size. Is there lterature regarding this problem? I see this in regard to the problem that in case means are in reality very similar (e.g. because of a ecological reason) then what is it with the statistical power, I mean would you need a humongous sample size to conclude that teh means are indeed similar due to a similar population? Thank you! $\endgroup$ – Vera Marya Apr 18 at 12:29
  • $\begingroup$ Note that we always assume a certain effect size against which we calculate power. If we knew the effect size, we would not need to do the actual data collection and analysis, because we would already know that the null hypothesis is false! And note that in a NHST, we cannot conclude that means are similar - we can only reject the null hypothesis or fail to reject it. $\endgroup$ – S. Kolassa - Reinstate Monica Apr 18 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.