Pilot sample sizes can yield inflated effect size estimates. After conducting a pilot I have found the 95% confidence interval on my effect to have a wide range due to the small sample size.

How does one account for this in a subsequent power analysis? E.g., consider a pre- to post-d score: d = 0.5, 95% CI [0.2, 0.8].

Do I use my best estimate (0.5)? Should I use the lower limit (0.2)? Or should I calculate a different confidence interval (say, 80%) and use the lower limit as to not grossly underestimate the size of the population effect?


This just happens to be a topic that has popped up in a few different areas lately:

This interactive tool that accompanies on a pub on the topic: http://pilotpower.table1.org/

This Lakens pre-print: https://psyarxiv.com/b7z4q

And this post from Andrew Gelman: http://andrewgelman.com/2018/03/20/purpose-pilot-study-demonstrate-feasibility-experiment-not-estimate-treatment-effect/

Of these, Gelman is the most direct in stating that the purpose of a pilot study isn't to estimate an effect at all.

Lakens seems to suggest that you will probably only use your pilot effect size if gives you a feasible sample size so you're setting yourself up for failure there to. He also gives a bit more advice if you still want to do a pilot:

  1. Use sequential analysis (i.e. your pilot study rolls into your actual study)
  2. OR use the lower bound of the 80% interval (as you are considering)

In other posts I've seen him argue to conduct a full decision analysis (requires a utility function) using your pilot data and then conduct value of information to determine whether a trial is worthwhile and if so how large it should be. This is consistent with how health economists think of accruing evidence for decision making.

The paper attached to the interactive tool (and Gelman) both give different versions of the argument that you should use your content expertise for similar interventions/phenomena to hypothesize either a realistic effect you want to detect or the minimally important effect (i.e. the smallest effect you would care about if true).

Frank Harrel argues that your sample size calc should capture your uncertainty in the true effect parameter so that you get a range of sample sizes.

The adaptive trial lit would say you identify an effect you care about, and then use the predictive distribution of your pilot to tell you how likely you would be to find a statistically significant result if you continued to recruit to a full sample (this is a version of sequential analysis).

There's probably more views on this as well, so sorry if all I've done is muddy the waters a bit.

Outside of hardcore trialists, industry, and stats methods though you'll find that most people don't pay much attention to any of this and just use their effect from their pilot (as long as it gives them a feasible sample size ;) )

  • $\begingroup$ +1 for a great answer. In fact, effect size estimates from small studies always have a large margin of error and "minimally important effects", as good as that sounds, are usually hard to determine, as human brains usually don't work with effect sizes (minimal important difference would be easier to determine, but effect size?). Usually there is no good basis for power calculation. However, my local ethics committee wants to see power calcuations no matter, what. If you happen to drop into those circumstances, you have to come up with something all the time. $\endgroup$ – Bernhard May 2 '18 at 11:45
  • 1
    $\begingroup$ Thank you for your detailed answer. From reading the Lakens pre-print and skimming over Gelmans post, I think the most feasible way forward is to use a one-sided 80% confidence limit on the estimated effect from my pilot, and use this value of d to calculate my main trial sample using a ‘safeguard power analysis’ (described in Perugini ncbi.nlm.nih.gov/pubmed/26173267). $\endgroup$ – pomodoro May 2 '18 at 13:10

Perhaps worth expanding on one of the points @Tdisher makes. In his article available here entitled "On the use of a pilot sample for sample size determination" Browne discusses the role of estimating the standard deviation

The abstract states

To compute the sample size needed to achieve the planned power for a t‐test, one needs an estimate of the population standard deviation δ. If one uses the sample standard deviation from a small pilot study as an estimate of δ, it is quite likely that the actual power for the planned study will be less than the planned power. Monte Carlo simulations indicate that using a 100(1 − γ) per cent upper one‐sided confidence limit on δ will provide a sample size sufficient to achieve the planned power in at least 100(1 − γ) per cent of such trials.

  • $\begingroup$ Yes, sorry this is very important also and is needed for a complete answer. Lots of focus on the mean and uncertainty in estimate of SD slips by un-scrutinized. $\endgroup$ – Tdisher May 2 '18 at 13:39
  • $\begingroup$ Is this essentially the same as what is being suggested in Perugini (ncbi.nlm.nih.gov/pubmed/26173267)? $\endgroup$ – pomodoro May 2 '18 at 13:51
  • 2
    $\begingroup$ It comes from a different tradition from Perugini as in health the mean difference should be established as the minimum clinically important difference and the pilot just estimates the variability. Cohen's measure is a combination of mean difference and variability. $\endgroup$ – mdewey May 2 '18 at 15:23
  • $\begingroup$ @mdewey would his cohens d account for uncertainty in estimate of sigma? My understanding is it wouldn't and that's what makes your answer so important? $\endgroup$ – Tdisher May 2 '18 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.