*Context*: I wrote this answer before the OP clarified that he is working with a large dataset, so his study (probably) has sufficient power. In my post I consider the more common case of a small study with a "significant finding". Imagine, for example, that the article under review presents an estimate of 1.25 in a domain where previous studies about related phonemena have reported estimates in the range [0.9, 1.1]. How does the article's author respond to the reviewer's request for a post-hoc estimate of power to detect an effect of size 1.25?
---
It's hard to make a reasonable argument that it doesn't matter if your study is underpowered. If a study has low power *and* the null hypothesis is rejected, then the estimate is likely to be biased upward. The less power, the more inflated the effect size estimate. Yes, you are lucky to get evidence against the null hypothesis but also likely to be over-optimistic. The reviewer knows this so he asks how much power your study had to detect the effect you detected.
It's also unreasonable to do post hoc power estimation. This is a much discussed topic on CV; see references below. In short – if your study was indeed underpowered – by doing post hoc power analysis you will compound the issue of an inflated effect estimate by also overestimating the power.
Okay, so enough bad news. How can you respond to the reviewer? Computing the power retroactively is pointless because your study is already done. Instead compute confidence interval(s) for the effect(s) of interest and emphasize estimation, not hypothesis testing. If the power of your study was low, the intervals will be wide (as low power means that we can't make precise estimates). If the power of your study was high, the intervals will be tight, demonstrating convincingly how much you have learned from your data.
*References*
J. M. Hoenig and D. M. Heisey. The abuse of power. *The American Statistician*, 55(1):19–24, 2001. <br>
A. Gelman. Don't calculate post-hoc power using observed estimate of effect size. *Annals of Surgery*, 269(1), 2019. <br>
[Do underpowered studies have increased likelihood of false positives?](https://stats.stackexchange.com/q/176384/237901) <br>
[What is the post-hoc power in my experiment? How to calculate this?](https://stats.stackexchange.com/q/430030/237901) <br>
[Why is the power of studies that only report significant effects not always 100%?](https://stats.stackexchange.com/questions/263383/why-is-the-power-of-studies-that-only-report-significant-effects-not-always-100) <br>
[Post hoc power analysis for a non significant result?](https://stats.stackexchange.com/questions/193726/post-hoc-power-analysis-for-a-non-significant-result) <br>
---
This simulation shows that "significant" estimates from underpowered studies are inflated.
``` r
library("pwr")
library("tidyverse")
# Choose settings for an underpowered study
mu0 <- 0
mu <- 0.1
sigma <- 1
alpha <- 0.05
power <- 0.5
pwr.t.test(d = (mu - mu0) / sigma, power = power, sig.level = alpha, type = "one.sample")
#>
#> One-sample t test power calculation
#>
#> n = 386.0261
#> d = 0.1
#> sig.level = 0.05
#> power = 0.5
#> alternative = two.sided
# Sample size to achieve 50% power to detect mean 0.1 with a one-sided t-test
n <- 387
# Simulate 1,000 studies with low power
set.seed(123)
reps <- 1000
studies <-
tibble(
study = rep(seq(reps), each = n),
x = rnorm(reps * n, mean = mu, sd = sigma)
)
results <- studies %>%
group_by(
study
) %>%
group_modify(
~ broom::tidy(t.test(.))
)
# Plot a histogram of the estimate effects for those studies where the null was rejected.
results %>%
# We are only interested in studies where the null is rejected
filter(
p.value < alpha
) %>%
ggplot(
aes(estimate)
) +
geom_histogram(
bins = 33
) +
geom_vline(
xintercept = mu,
color = "red"
) +
labs(
x = glue::glue("estimate of true effect {mu} in studies with {100*power}% power"),
y = "",
title = "\"Significant\" effect estimates from underpowered studies are inflated"
)
```
<sup>Created on 2022-04-30 by the [reprex package](https://reprex.tidyverse.org) (v2.0.1)</sup>