1
$\begingroup$

Journals and reviewers increasingly ask authors to systematically report sensitivity power analyses. I know that a sensitivity power analysis allows you to determine the minimum effect size that the study was sensitive to for a certain level of power, based on the sample size recruited and the alpha level specified.

But I'm a stats noob. Can someone explain this in lay terms? What's the rationale behind this push?

Can you give me examples of interpretations for each of the following scenarios?

  1. Your observed effect size is greater than the predicted minimum effect size, and you found a significant effect
  2. Your observed effect size is greater than the predicted minimum effect size, and you found a null effect
  3. Your observed effect size is smaller than the predicted minimum effect size, and you found a significant effect, and
  4. Your observed effect size is smaller than the predicted minimum effect size, and you found a null effect.

What can we conclude in each case?

$\endgroup$
0
$\begingroup$

Regarding your first question: What is the rationale for a sensitivity power analysis?

In many real-world situations anything may have an effect. The question

"Does X have an effect on Y?"

is therefore insensible.

"Does X affect Y to speak of?"

is most often a better question.

A completely fictitious example: certain sugar pills may be beneficial against cancer and have a specific effect beyond placebo, yet, that effect may be so small (e.g. life-prolonging by mere seconds) that it is irrelevant (let alone unmeasurable with even the biggest sample size) and canceled out by even the smallest everyday event (e.g. by having a cup of coffee). Providing evidence for a statistically significant "above zero" effect of these pills would be futile. Trying to show "an effect beyond practical significance" (e.g. "doesn't prolong life by at least a day") would be more sensible.

Daniel Lakens puts it like this:

the sensitivity power analysis signifies "the smallest effect size you care about".

When planning a study, researchers should first determine the minimum important effect size for their research question, based on practical and/or theoretical considerations. Now, what is "minimum important" you ask? This strongly depends on the field of research and the phenomenon in question and is a matter of consensus. In the medical field, "minimally clinically significant effect sizes" exist for many indications, often focusing on patient quality of life (and often these are subject of heavy discussion). In fields like particle-physics, even the most minuscule effect above noise may bear a grand revelation (but that is not my area of expertise). If your research outcome has real-world consequences cost-benefit may factor in, and so on, but this is going beyond your question... In an ideal world, the researcher would perform a regular power analysis as a second step and determine the sample size required to reliably detect the effect size of minimal importance (where the "reliability" is selected in terms of alpha and beta levels). However, in real world there are often practical constraints to the sample size. The researcher may face limited resources (manpower, time, money...), or a limited available study population. In these cases she can at least use sensitivity analysis to calculate the minimum detectable effect size and use it in discussion, to compare it against both the observed effect size and the minimally interesting effect size.

And there we are at your second question:

  1. Your observed effect size is greater than the predicted minimal detectable effect size, and your test indicates a statistically significant effect

You reject the null-hypothesis and claim evidence in favor of the effect. You can now focus on discussing how far your observed and minimum detectable effect sizes were above or below the lower bound of practical/theoretical interest (minimal detectable/interesting effect size), focusing on practical significance (this is more important than p>.05 or "statistical significance"). You may argue that your study was over-/under-powered depending on whether the minimum detectable effect size was below/above the threshold of practical/theoretical interest. You may argue that the relevance of your finding was high or low, depending on whether the observed effect size was above or below the threshold of practical/theoretical interest, respectively.

  1. Your observed effect size is greater than the predicted minimal detectable effect size, your test result is not statistically significant

This case should not occur if power analysis and statistical testing were based on the same alpha and n. However, power analysis is often performed before the study. The realized final sample may be smaller than planned. Further, if you expressed your minimal detectable effect size in terms of non-standardized effect sizes (so NOT Cohen's d, eta^2, r etc.), the variance in measurements factors in (in case of repeated measurements the correlation between measurements, as well). You may have assumed a lower SD in your power sensitivity analysis (which is based on standardized effect sizes) than what was observed in the sample. You can discuss this result, and additionally provide an updated power sensitivity analysis demonstrating the "realized" sensitivity of the study. >> see case 4

  1. Your observed effect size is smaller than the predicted minimum detectable effect size, your test indicates a statistically significant effect

In other words, our observed effect size was smaller than the smallest "true" effect size that your study was set to detect "reliably", nevertheless the statistical test indicated statistical significance. This scenario is possible, especially when a high beta-value was chosen. You reject the null-hypothesis and claim evidence in favor of the effect. You should discuss that despite evidence for an effect, the observed effect size was below the bound of what your study was set to detect "reliably". This may put the reliability of results in question, depending on the standards you've set. Beyond that, see Case 1 for discussing practical/theoretical significance.

  1. Your observed effect size is smaller than the minimal detectable effect size, and you found a null effect.

    Your observed effect size was below the smallest "true" effect size that your study was set to detect "reliably" and no evidence for an effect was found. Here, the results of your sensitivity analysis are most interesting: A negative test result may equally mean "no effect", or "indeed an effect, but too small to be certain, given this sample size". The sensitivity power analysis can help you to better interpret these results. It depends on what your minimal detectable effect size represents:

    • If the minimal detectable effect size is above the minimum of practical/theoretical interest, your results are inconclusive and your study was underpowered. A true effect with an effect size of interest may (or may not) exist, but your study did not provide enough information to be reasonably certain.
    • If the minimal detectable effect size was at or below the threshold of minimum practical/theoretical interest you can argue that your study was powered sufficiently and that the true effect in question is likely smaller than what is practically/theoretically relevant.

In summary, it is good scientific practice to both report minimum detectable effect size and the minimum effect size of practical/theoretical interest, regardless of your test outcomes. These metrics help to interpret your results by focusing on effect sizes instead of statistical significance and therefore bridge the argumentative gap between "statistical" and "practical" significance.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks! In this case, is it still relevant to report sensitivity power analyses when we did not get null findings (i.e., only significant findings)? It was my understanding that it was not important in this situation, but it seems like journals ask for sensitivity power analyses regardless of results, thus my confusion. $\endgroup$ – RemPsyc Feb 6 at 18:01
  • $\begingroup$ Ok, edited the answer above, accordingly. The problem lies in the focus on statistical significance rather than effect sizes. $\endgroup$ – mzunhammer Feb 7 at 10:51
  • $\begingroup$ Amazing, this is what I was looking for! For guidance when reporting the analysis in a paper. Thanks! $\endgroup$ – RemPsyc Feb 7 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.