0
$\begingroup$

I have an experiment which is examining thickness of hair before and after a certain treatment. I have 2 different treatments. I calculated the effect size in R of the treatments and found the confidence interval of the effect size to include 0, meaning the difference in thickness of hair after a treatment was not affected.

I would like to be able to calculate how many more participants I would need for sample size needed to detect a difference ie. to not span 0, and I have been told I can use length of confidence interval. I've tried to figure it out but I am still confused.

I had 30 participants in my pilot study, and would like at least to see an effect size of 0.3. I am just looking at differences within participants.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

To answer such questions you would need to do a power analysis. And to do power analysis you will need 4 values:

  • mean difference between groups
  • standard deviation (pooled from two groups)
  • sample size
  • significance level (typically 0.05)

If you have these then you can use R with the pwr library to calculate the power of the test to detect the difference between groups under your specified parameters:

library(pwr)
pwr.t.test(n=15, d=0.3/1, sig.level=0.05)

Here n is sample size in of one group, d is Cohen's d which is difference between means divided by standard deviation and sig.level is your wanted significance level (probability of false positive result).

Note that I could not find standard deviation from your post - so I used 1 instead. You would have to change this number according to your data.

The function will output the following:

 Two-sample t test power calculation

              n = 15
              d = 0.3
      sig.level = 0.05
          power = 0.1246978
    alternative = two.sided

You might be interested in calculating the sample size needed to achieve a reasonable good power (by default 0.8 is considered good in most cases). You can do this by removing the sample size from the function arguments and adding the wanted power instead:

pwr.t.test(d=0.3/1, sig.level=0.05, power=0.8)

     Two-sample t test power calculation

              n = 175.3847
              d = 0.3
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

This means that to have 80% chance of detecting the 0.3 difference (with 1 as a standard deviation) you would have to have 175 samples per group.

Most likely is that in your case the standard deviation is lower - so you will have to plug it in and see.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This is the advice I got- If you know that an interval spans zero, the interval will give you an idea of how large a treatment effect could be to still be consistent of your dataset. Find out the length of the interval that allows you to find a difference is the true effect was a specified size ie. how many more participants would you need for your CI to not include zero? $\endgroup$
    – mahyea
    Commented Mar 1, 2017 at 20:09
  • $\begingroup$ Confidence interval is constructed from standard error. And standard error is sd/sqrt(n) where sd is standard deviation and sqrt(n) is a a square root of your sample size. From this formula you can infer that to shrink the confidence interval by half you would need to increase your sample size 4 times. But of course I must note that this procedure will not guarantee you any significant result. For example - your real effect size might truly be zero. $\endgroup$
    – Karolis Koncevičius
    Commented Mar 1, 2017 at 20:21
  • $\begingroup$ This is what I am confused about- my effect size CI already contains zero, and the estimate of the effect size is something like 0.003, so how am I to adjust it to the point where it doesn't include zero? $\endgroup$
    – mahyea
    Commented Mar 2, 2017 at 15:41
0
$\begingroup$

If you are using R, then you should use the pwr.t.test to calculate your sample size (n), by setting your effect variable (d) equal to 0.3.

You can calculate as follows:

pwr.t.test(n = , d = , sig.level = , power = , type = c("two.sample", "one.sample", "paired"))

You might find this link helpful: http://statmethods.net/stats/power.html

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is the advice I got- If you know that an interval spans zero, the interval will give you an idea of how large a treatment effect could be to still be consistent of your dataset. Find out the length of the interval that allows you to find a difference is the true effect was a specified size ie. how many more participants would you need for your CI to not include zero? $\endgroup$
    – mahyea
    Commented Mar 1, 2017 at 20:16
  • $\begingroup$ Well, think of it this way. The more observations you have, the better (at least in theory) since it follows the law of large numbers where the difference between the expected and actual value drops to zero. However, let's say that n=100 was sufficient for this to happen. Would it make sense to collect 1000 observations? Of course not. So, you are looking for the maximum number of observations you would need to collect for the effect to be the same. $\endgroup$
    – Michael Grogan
    Commented Mar 1, 2017 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.