Interpreting significant effect sizes smaller than those used in sample size calculation

Question

Lately, I've been reading a lot about power calculation, effect size and sample size of an experiment, but recently this question arose in my head and I do not know what to do of it and would really like your thoughts about it.

Suppose I have a simple two-sample t-test I'd like to carry out and I want to detect a big effect, say Cohen's d = 0.8, and the minimum sample size in each group is 26 (alpha=0.05,beta=0.2 and using R's pwr package), so I use 52 experimental subjects in total as prescribed. By doing so, I can be sure that I have enough statistical power as to detect a difference in means that is "big" (Cohen's d >= 0.8).

The question then is: what should I think about getting a significant result (p < 0.05) in a difference in means smaller than d=0.8 ? How should I/would you interpret this finding and report/discuss it in a paper?

Answer 1

Generally, the p-value can be less than 0.05 even if the observed value of the estimated parameter is much smaller than the postulated value. There is a simple formula for the smallest observed value you need to have significant result. Of course, that assumes all your assumptions are correct and that the estimate is normally distributed. If you are doing two-sided testing, with the assumptions you stated it can be 70% as large as the postulated value.
In R, you can find that using this formula

qnorm(0.975)/(qnorm(0.975)+qnorm(0.8))

If one-sided testing, then 66% as large as postulated.
In either case, when you reject the null hypothesis, the confidence interval will exclude 0 but will include the postulated value. You would discuss it in a paper by reporting the observed estimated value and the confidence interval. The estimate is always just an estimate and you never know precisely what the parameter is. If it is important to prove that Cohen's d is truly larger than some value like 0.8, then make the null hypothesis d=0.8 not d=0. Find the sample size needed for a postulated d of 0.9 (or something else bigger than 0.8). Then, you will have a 80% chance of having not just the estimate but the entire confidence interval above 0.8 if the assumptions are correct. If you assume the true d is 0.8, no matter how big you make the sample size, there is only a 50% chance that the observed d will be larger than 0.8 (and 50% it will be smaller).