2
$\begingroup$

Suppose one would like to test some new hypothesis, for which there are no previous data available. To estimate the needed sample size, one should do a power analysis. Since there are no previous data available, the power analysis can only be carried out by conducting a pilot study first. How does one choose the sample size for a pilot study?

The field is neuroscience, where sample sizes are generally small due to financial and other constraints (cost per measured subject is easily in the thousands).

If I'd like to compare two groups, would a pilot study with 4+4 participants be enough to run a power analysis? How does the sample size of the pilot study affect the reliability of the power analysis?

Are the there any techniques (e.g. bootstrap?) that I could use to improve the power analysis?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

A pilot study is a good idea if an experimenter does not have recent historical data that can be used to estimate sigma. The effect that you want to detect should be the smallest effect that has practical significance. You don't need a pilot study to determine that. Pilot studies are generally small. You want to have enough observations so that you can estimate sigma with reasonable precision, as a poor estimate will undermine the hypothesis test.

Improve this answer
$\endgroup$
0
$\begingroup$

If you want to compare two groups, you can do a power analysis without a pilot study! See for example: http://www.ats.ucla.edu/stat/stata/dae/t_test_power2.htm

Conversely, if your analysis should go beyond a simple treatment-control comparison, you could simulate data and analysis: Power analysis for ordinal logistic regression

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ Power analysis needs an estimate of effect size. Without any previous data, one does not know it. $\endgroup$
    – mmh
    Commented Jun 20, 2015 at 13:07
  • $\begingroup$ Yes, but I mean you can do an hypothesis on the expected effect size! $\endgroup$
    – stochazesthai
    Commented Jun 20, 2015 at 13:15
  • 1
    $\begingroup$ Moreover, if the sample size is 4 per group the effect size needed to achieve a 0.8 power should be huge... $\endgroup$
    – stochazesthai
    Commented Jun 20, 2015 at 13:16
  • 1
    $\begingroup$ If you are studying something new, how could you have an expected effect size? It would be just pure guessing without a pilot study. Though, I don't know how to answer the question. $\endgroup$
    – mmh
    Commented Jun 20, 2015 at 13:16
  • 1
    $\begingroup$ Well, you are designing an experiment so that IF you hypothesize a certain effect size, you need a sample size pf N to achieve a power of 0.8 and a significance level of 0.95. I mean when you are designing an experiment for a new drug that should reduce the mortality you don't have previous studies but you guess an effect size in order to design the experiment. $\endgroup$
    – stochazesthai
    Commented Jun 20, 2015 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.