Suppose we have an AB test running for enough time to observe an effect size of at least 3% on certain variable X. When checking the results, we obtain that the p-value is below the significance level (< 0.05) but the effect observed during the experiment is just 2%.
To me, it looks like we don't have enough statistical power to judge the results as significant even when we have a significant p-value. How do you judge these results in this situation? Can we say that the results are significant having that the observed effect size is smaller than the one the test was able to detect at least 80% of the time?
Is there any post-hoc analysis that could be ran on this situation?
I've been looking at some papers but some of them contradict each other when approaching to post-hoc analysis.
Thank you in advance for your help!