Is the post-hoc observed effect size redundant with the p-value, just like post-hoc observed power is?

Question

Assume for simplicity sake that we are running a 2-sample t-test on 2 normally distributed populations, from which we collect 2 i.i.d samples. (for a start; we can expand later to more complex cases). We know that the post-hoc observed power of a study is fully determined by the observed p-value. See here, or here, and multiple references on this site.
Then, post-hoc, we have the planned $\alpha$ (.05 by tradition), actual sample size $N$, and p-value $p$. But from $p$ we can get to the observed power ($1-\beta'$), as shown in the links. And we have the observed standard deviation ($sd$). And I can compute the observed effect ($\delta=(xbar_1-xbar_2)/sd$).
Let me know repeat this, with 2 new samples (same sample size, same normal distributions, same variances, but different "true" effect). I get a different p-value, observed power $(1-\beta$) and observed effect $\delta$. I repeat the same many more times. Is it correct to think that there is a monotonously decreasing funtion of the $\delta$ values vs. the p-values? (just as the links show for power vs. p). Or is post-hoc effect-size a random variable? In this case, what might be its distribution?

Answer 1

Yes. There is a monotonic relationship between the observed p-value and observed $\delta$, assuming fixed sample size, null hypothesis, and test type. That means that you can calculate the p-value from $\delta$, sample size, null hypothesis, and test type, just as you can calculate the $\delta$ from the p-value, sample size, null hypothesis, and test type.

You might find it helpful to consider, for example, the nature of the test statistic for a Student's t-test $$t=\frac{(\bar{x}_1-\bar{x}_2)-\delta_0}{SED}$$ where $SED$ is the standard error of the difference and $\delta_0$ is the null hypothesised value of the $\delta$. (I much prefer specifying the null hypothesis in that way because of its clear connection to the observed delta and so that it is clear that it need not be zero.) That $t$ value is determined by your $\delta$ along with sample size. There is a one to one relationship between any observed value of $t$ and the p-value assuming fixed sample size and null hypothesis and test tails.

Answer 2

Well, since this question nagged me, I decided to try a quick simulation and answer it myself. I ran 1000 1-sample t-tests, with samples from $N(0,1)$. I then just collected p-values, observed power, and observed effect.
This is the graph I got for 1000 replications, with an effect of .1, for the relationship between p-value and observed power. Just as in Hoenig and Heisey's paper. This is then the result for observed effect size.
And both of them on the same graph One can change the true effect (even set it to 0), and the curves do not change, at all (except that for large effects, we get a lot fewer large p-values, as expected, so the graph gets "censored".

This now begs a new question: since post-hoc power analysis is, at best redundant, at worse misleading (see Hoenig & Heisey, etc), should we not also say that post-hoc effect sizing is equally redundant, and probably misleading? (to keep things clear my question is only for power/effect calculations which are based solely on the observed data, not for prospective calculations, for possible future studies)