# Is the post-hoc observed effect size redundant with the p-value, just like post-hoc observed power is?

Assume for simplicity sake that we are running a 2-sample t-test on 2 normally distributed populations, from which we collect 2 i.i.d samples. (for a start; we can expand later to more complex cases). We know that the post-hoc observed power of a study is fully determined by the observed p-value. See here, or here, and multiple references on this site.
Then, post-hoc, we have the planned $$\alpha$$ (.05 by tradition), actual sample size $$N$$, and p-value $$p$$. But from $$p$$ we can get to the observed power ($$1-\beta'$$), as shown in the links. And we have the observed standard deviation ($$sd$$). And I can compute the observed effect ($$\delta=(xbar_1-xbar_2)/sd$$).
Let me know repeat this, with 2 new samples (same sample size, same normal distributions, same variances, but different "true" effect). I get a different p-value, observed power $$(1-\beta$$) and observed effect $$\delta$$. I repeat the same many more times. Is it correct to think that there is a monotonously decreasing funtion of the $$\delta$$ values vs. the p-values? (just as the links show for power vs. p). Or is post-hoc effect-size a random variable? In this case, what might be its distribution?

• $\delta'$ is the observed effect, which of course won't change, so I assume this is a typo and you are asking about the relationship between $\delta$ (not $\delta'$) and $p$, yes? My go-to reference for post-hoc power is Hoenig & Heisey (2001, The American Statistician). Commented Apr 18 at 6:19
• @StephanKolassa. To clarify, if I were to now run another 2-sample t-test (with a different actual effect, sd, etc.), but same sample size, I would get another p-value, $\delta'$, and observed power $(1-\beta')$. We know that p and $(1-\beta')$ are deterministically related (see links provided). Is there a similar relationship between p and $\delta'$? I will clarify the question. Commented Apr 18 at 16:23
• @StephanKolassa, I re-read the paper you indicated, and while it talks about some forms of post-hoc effect size estimation, it does not answer my question, nor does it provide similar evidence as it does against post-hoc power estimations... Unfortunate Commented Apr 18 at 16:51

Yes. There is a monotonic relationship between the observed p-value and observed $$\delta$$, assuming fixed sample size, null hypothesis, and test type. That means that you can calculate the p-value from $$\delta$$, sample size, null hypothesis, and test type, just as you can calculate the $$\delta$$ from the p-value, sample size, null hypothesis, and test type.

You might find it helpful to consider, for example, the nature of the test statistic for a Student's t-test $$t=\frac{(\bar{x}_1-\bar{x}_2)-\delta_0}{SED}$$ where $$SED$$ is the standard error of the difference and $$\delta_0$$ is the null hypothesised value of the $$\delta$$. (I much prefer specifying the null hypothesis in that way because of its clear connection to the observed delta and so that it is clear that it need not be zero.) That $$t$$ value is determined by your $$\delta$$ along with sample size. There is a one to one relationship between any observed value of $$t$$ and the p-value assuming fixed sample size and null hypothesis and test tails.

• thanks for teh answer. This matches my intuition, but I was surprised to not see that mentioned in literature? This then begs another question. Now that most statisticians know that asking for post-hoc power is, at best, redundant (with the p-value), many editors/reviewers are asking for effect size. But effect size is just as redundant, and may lead to a false sense of confidence ("my effect was large, so my p-value is even more significant"). So does it make sense to even compute, and publish post-hoc effect sizes, as they are fully predicted by the p-value? Commented Apr 19 at 21:42
• Um, hem, hmmm. When did you last see editors/reviewers accused of good sense? Seriously, I think that simple measures of effect size are important even if they are encoded into p-value or confidence intervals etc. However, my special extra to that is that the specification of observed effect size should be explicitly described in relation to the effect size that might matter biologically or scientifically. Do not fall into the trap of thinking that statistical effect size is important to science! Commented Apr 19 at 22:10
• The other thing that is worth considering is that many journals (and statisticians) suggest that p-values be expressed only as less thans like P<0.05, and in that form they do not tell you much about even the statistical effect size. Commented Apr 19 at 22:12
• +1, but I'm wondering something about "you can calculate the δ from the p-value, sample size, null hypothesis, and test type": assuming we're conducting a two-sided test, I think we can't find the direction of the effect size just from the p-value and the sample size. But I may be mistaken. Commented May 3 at 7:23
• @J-J-J Yes, I think you are correct. I guess I was thinking one-sided. I far prefer one-sided tests because they are much more closely aligned with sensible notions of evidence. Details here: arxiv.org/abs/1311.0081 Commented May 3 at 20:55

Well, since this question nagged me, I decided to try a quick simulation and answer it myself. I ran 1000 1-sample t-tests, with samples from $$N(0,1)$$. I then just collected p-values, observed power, and observed effect.
This is the graph I got for 1000 replications, with an effect of .1, for the relationship between p-value and observed power. Just as in Hoenig and Heisey's paper. This is then the result for observed effect size.
And both of them on the same graph One can change the true effect (even set it to 0), and the curves do not change, at all (except that for large effects, we get a lot fewer large p-values, as expected, so the graph gets "censored".

This now begs a new question: since post-hoc power analysis is, at best redundant, at worse misleading (see Hoenig & Heisey, etc), should we not also say that post-hoc effect sizing is equally redundant, and probably misleading? (to keep things clear my question is only for power/effect calculations which are based solely on the observed data, not for prospective calculations, for possible future studies)

• I can see how post-hoc power can be misleading as well as redundant, but how do you think that the observed effect size is misleading? (I am assuming that is what you mean by "post-hoc effect sizing"). Surely reports of results should include a statement of the effect size (and, ideally, argument regarding the importance of that effect size). Commented Apr 25 at 21:16
• Well, that is the whole point of my question. If, as my simulation demonstrates, the effect size is entirely predicted by the p-value, then the effect size does not add any new information, and can be misleading. If you get a small p, then you will get a good power, and a large effect: these are not independent pieces of information which strengthen your conclusion. They are redundant: finding confirmation in the large effect is misleading... The only meaningfull piece of information is the p-value. The rest is a direct consequences, and does not "further" confirm your rejection of the null... Commented Apr 26 at 1:53
• ... But you could argue that, among the p-value, effect, and power, any 1 piece of information out of the 3 is as justifiable as the other. I would agree with this. But then, trolling out the other 2 as "further evidence" (that the study was welll powered, or that the effect was large, that the p is incompatible with the Null) is misleading. They are deterministically correlated to the first you chose to report. Big deal that you got a large effect; that is because you got a low p-value; and the same can be said for the other 2 pairs... Commented Apr 26 at 1:58