Stat sig results on underpowered experiment I ran an experiment that was powered enough to detect lifts of 1% on my 2-sample test.
Make it really simple:
I have two normal distributions and want to compare the means, so I ran a 2-sample t-test over a lift of 0.5%.
So they are two scenarios:

*

*P-value above 0.05, I reject Null and assume the 0.5% is random. BUT, to validate that, it's better to run the XP again but this time powered enough to detect if the 0.5% is significant or not.


*The 0.5% appears significant. What does this means? From my understanding, this effect will be overestimated, because since it's not powered enough, only "big" impacts will be detect. But even overestimated, you can conclude there's some effect out there that is not random.
What would you do if you face scenario 2?
 A: You don't provide details (what is powered enough?), so it's necessary to make assumptions.
I'll assume that when you say the effect 0.005 appears significant, you mean the observed effect is $\hat{\mu}$ = 0.005 and the lower limit of the (1-α)% confidence interval is above 0. I'll also assume that the test is one-sided (the aim is to increase lift) and the power at the alternative hypothesis $\mu_{\text{A}}$ = 0.01 is (1-β)% (since you don't specify the number).
Power of (1-β)% means that, under the alternative hypothesis $\mu_{\text{A}}$ = 0.01, if you were to repeat the experiment, (1-β)% of the (1-α)% confidence intervals for $\mu$ will be entirely above 0.
In some of those hypothetical replications the (1-α)% confidence interval will exclude 0 and the estimated effect will have a value $\hat{\mu}$ < $\mu_{\text{A}}$, which is what you observe in your experiment. So you have indirect evidence for an increase in lift and if you reject the null hypothesis the probability of false discovery (type I error) is 5%. That's not evidence in favor a "data-derived" alternative hypothesis that the true effect size is 0.005.
What I would do: Rather than calculate the sample size n to achieve power (1-β)% at a specific alternative $\mu_{\text{A}}$, I would pick n to control the margin of error, i.e., estimate $\mu$ to within a margin of error $\pm \delta$ with (1-α)% confidence. And I'll pick $\delta$ that's relevant for the business.
What I wouldn't do: I won't be wondering what's the post-hoc power at the observed effect whether $\hat{\mu}$ is smaller or bigger than the $\mu_{\text{A}}$ value in the original power calculation.
PS: "Powered enough" doesn't bring to mind power of 99% but just in case, here is a subtle point. If the power is very high, we expect a very small p-value under the alternative. If the observed p-value is just below the significance threshold, say p = 0.04, the data look "unusual" under both the null (the p-value is small) and the alternative (we expected an extreme p-value if $H_{\text{A}}$ were true).
See Why p-values should be interpreted as p-values and not as measures of evidence
