If we shouldn't do post hoc power calculations, are post hoc effect size calculations also invalid?

Question

We have posts on here discussing why post hoc power calculations are flawed.

What about post hoc calculations of the effect size that we could catch? That seems logically equivalent. If we are not supposed to calculate the power to detect an effect of $1$, then we should not calculate the effect we can detect with $80\%$ power.

(Perhaps this warrants a separate question, but sample size and $\alpha$-level also play into the equation. Is it invalid to do a post hoc calculation of the sample size or $\alpha$? (I am thinking of how R's pwr package will calculate one if you give the others, something like what I posted when I first joined Cross Validated.) But I mostly want to talk about post hoc calculations of effect size.)

EXAMPLE

We have observations from two groups and perform a t-test of their means that gives $p=0.1626$: insignificant at $\alpha=0.05$.

We want to declare equivalence of the groups, but the subject matter experts (SMEs) are unsure how small of an effect size warrants equivalence. "Tell us what kind of effect you can detect," they tell me. "We'll come back and say if that effect is important."

I know that we like to operate with the standards of $\alpha=0.05$ and a power of $80\%$, so I use the sample sizes, $\alpha$-level, power, and observed variances to calculate the effect size I can detect. I tell the SMEs, "We can detect an effect size of $0.25$."

If I were to flip the problem around to have an effect size of interest, say $0.3$, and calculate that we have $90\%$ power to detect such a difference, that seems like an invalid post hoc power calculation. It seems like the post hoc effect size calculation is just as invalid.

EDIT

Really, it comes from a paper by Rebecca A. Betensky: "The p-Value Requires Context, Not a Threshold". She shows what to do if we declare an effect size of interest. I can imagine her method being turned around to have me answering what kind of effect we can detect.

Answer 1

The problem is in the use of the "post-hoc effect size," not that its calculation is invalid. A "post-hoc effect size" is fundamentally an estimate of population parameters (e.g., mean difference between two groups and a standard deviation, not a standard error!) whose precisions might be affected by study design but aren't otherwise determined by study design. After all, using a "post-hoc effect-size" estimate from a pilot study to design a definitive study is good practice.

One problem, addressed in many threads here as you note, is that a post-hoc power calculation based on the "post-hoc effect size" is meaningless. As Russ Lenth puts it so simply:

You’ve got the data, did the analysis, and did not achieve “significance.” So you compute power retrospectively to see if the test was powerful enough or not. This is an empty question. Of course it wasn’t powerful enough – that’s why the result isn’t significant. Power calculations are useful for design, not analysis.

In your example, there is similarly no error in calculating the "post-hoc effect size." The error is in your clients' trying to use that "post-hoc effect size" to reverse-engineer an "insignificant" null-hypothesis test into a post-hoc equivalence test. Null-hypothesis tests and equivalence tests are fundamentally different. Attempts at post-hoc equivalence tests aren't just meaningless; they are misleading. This page provides more details and a literature reference. In particular, Walker and Nowacki emphasize:

The determination of the equivalence margin, $\delta$, is the most critical step in equivalence/noninferiority testing... the value of the equivalence margin should be determined before the data is recorded. This is essential to maintain the type I error at the desired level...

Using a traditional comparative test to establish equivalence/noninferiority leads frequently to incorrect conclusions. The reason is two-fold. First, the burden of the proof is on the wrong hypothesis, i.e., that of a difference... the risk of incorrectly concluding equivalence can be very high. The other reason is that the margin of equivalence is not considered, and thus the concept of equivalence is not well defined.

As always, proper specification of the question should precede the study design and data analysis. Changing the hypothesis after the results are obtained is the problem.