4
$\begingroup$

Power is the probability of rejecting the null hypothesis when the null is truly False. It depends on effect size, sample size and the signifance level.

If I want to estimate the sample size I need to run my experiment: I need to determine: effect size, signifance level, power. Power is typically 80% and alpha is typically 0.05. But how do I determine the effect size?

Thanks for any help!

$\endgroup$
4
$\begingroup$

It will depend very much on your question, and your field (or even subfield). As an example, one day in a graduate class, an instructor asked if we thought a relative risk of 1.25 (or something like that) was a "big effect".

  • The environmental epidemiologists all raised their hands. Lawsuits would be filed, and EPA regulations passed on things with a RR of 1.25
  • The cancer epidemiologists mostly raised their hands. The "low-hanging fruit" for cancer causes has already been found for the most part, so something with that much of an increase in risk is probably a big deal.
  • The infectious disease epidemiologists on the other hand, mostly responded with a shrug - we're often used to dealing with much larger effect sizes.

So it depends. You should choose an effect size based on some criteria - is there an initial study that suggests something? A meta-analysis?

If there isn't, you may be well served by calculating power over a larger range of variables. For example, this is a figure from a power-calculation I performed:

enter image description here

Here, I wasn't sure of either the effect measure, or the ratio of exposed to unexposed individuals in the data, so I varied those to make sure that, under a number of different circumstances that my sample size was sufficient.

$\endgroup$
3
$\begingroup$

It depends whether you aim for superiority, equivalence, or non-inferiority.

In most typical scenarios we aim for superiority.

Then the effect size would be the mininal effect that would be clinically relevant or meaningful, and you need subject competence on this.

Imagine you want to compare a new and likely more potent drug to lower blood pressure to the one which is standard of care. Let us assume that a difference less than 0.3 mm Hg (eg 0.001 mm Hg) would be irrelevant for patients and physicians (even if statistically significant!). Then you'll aim for an effect size of at least 0.3 mm Hg (say, 0.55 mm Hg). Remember that you also need to factor in the variability of the phenomenon.

Take note that in a superiority scenario alpha will be 2-tailed.

In a hypothetical case where you compare two blood pressure lowering drugs in a randomized trial with 1:1 allocation, you assume an effect size with mean difference of 0.55 mm Hg, a standard deviation in each group of 0.9 mm, and aim for 0.8 power and 0.05 2-tailed alpha within a superiority framework. According to the computations below (samplesize package in R), you will need 88 patients (44 per group).

n.ttest(power = 0.8, alpha = 0.05, mean.diff = 0.55, sd1 = 0.90, k = 1,
design = "unpaired", fraction = "balanced", variance = "equal")

$`Total sample size`
[1] 88

$`Sample size group 1`
[1] 44

$`Sample size group 2`
[1] 44
$\endgroup$
2
$\begingroup$

The effect size will come from subject matter considerations (prior studies, theory, knowledge of similar variables and so on).

One of the considerations is what would be a smallest "substantive" effect, one big enough to be considered important. You are trying to identify an answer to the question "what is the effect size that you want to have an 80% chance* of picking up?".

* (or whatever other power you put in to the calculation; there's nothing special about 80% even if it's widely used in psychology).

To answer that question, you'd at a minimum need to figure out what sort of effect size people in that subject matter area might care about. That will depend on many things.

If you choose a large effect size and base your sample size calculation on that, any smaller effect sizes will be less likely to be identified, so you risk having too small a sample size to spot it.

So you also need to have an idea (from subject matter knowledge, as above) how large an effect of this kind might tend to be (others with similar subject matter knowledge will presumably also consider it a likely effect size or can be convinced of it from an argument that relies on that shared understanding of the subject).

Guessing much to small will result in unnecessarily huge sample sizes, and guessing much too large risks wasting the study.

Note that Cohen (1992) "A power primer" said:

My intent was that medium ES represent an effect likely to be visible to the naked eye of a careful observer. (It has since been noted in effect-size surveys that it approximates the average size of observed effects in various fields. ) I set small ES to be noticeably smaller than medium but not so small as to be trivial,

... so Cohen's "small" effect size would relate to the first thing I mentioned (an effect size "people will care about"), while (by his parenthetical comment) his "medium" one is closer to typical effect sizes in that subject area (the second criterion I discussed); this will typically be larger than effect sizes people care about (otherwise they're in a subject area where they won't find many of the actual effects that exist worth caring about).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy