P value as a measure of effect size?

Question

Why is the p-value not a measure of effect size, i.e a lower p-value having a higher Cohen's D or mean difference? Say for instance I'm doing multiple t-tests or mann whitney u tests and using a p-value correction, does the lower the p value among those I've calculated indicate a larger effect size? If not, why not?

Edit: I understand the case when sample sizes are different, power is higher with larger sample and as a result likelihood to detect a smaller effect is higher, but I'm referring to the situation when one uses the same sample size and does multiple tests. For example, say I am comparing heights of Golden State Warriors with heights of 10 other teams or even comparing two other teams and sample size is same for each team, would lower p-value in the tests indicate higher effect?

Answer 1

Your intuition is correct here --- although the p-value is not used as a measure of effect size, you are correct that in some tests, for a fixed sample size the distribution of the p-value is monotonically related to the effect size, and thus is implicitly a transformed estimator of the effect size. Generally speaking, a larger effect size (further from the null hypothesis) manifests in a smaller p-value. In many cases it is possible to establish a stochastic dominance result to this effect.

---

Example - One sample two-sided Z-test: To illustrate this phenomenon, consider the simple case where we have IID normal data and we take a one-sample Z test of the population mean $\mu \in \mathbb{R}$ with known population variance $\sigma = 1$. (This is not a very realistic scenario, but it is the simplest version of the hypothesis test for a mean, so it is useful for illustrative purposes.) Taking a two-sided test with null hypothesis $H_0: \mu = \mu_0$ we have the test statistic:

$$Z(\mathbf{x}_n) = \sqrt{n} \cdot (\bar{x}_n - \mu_0),$$

with the corresponding p-value function $p(\mathbf{x}_n) = 2 \cdot \Phi(-|Z(\mathbf{x}_n)|)$. If the true mean is $\mu$ then the absolute value of the test statistic has a folded normal distribution:

$$|Z(\mathbf{X}_n)| \sim \text{FN} \Big( \sqrt{n} \cdot (\mu - \mu_0), 1 \Big).$$

Now we apply the standard rules for transformations of probability density functions to obtain the the p-value density function. The transformation $p = 2 \Phi(-z)$ has inverse $z = - \Phi^{-1} (p/2)$, so we get:

$$\begin{align} f(p) &= f(z(p)) \times \Bigg| \frac{dz}{dp} \Bigg| \\[6pt] &= \text{FN} \Big( - \Phi^{-1} (\tfrac{p}{2}) \Big| \sqrt{n} \cdot (\mu - \mu_0), 1 \Big) \times \Bigg( \frac{1}{2} \cdot \frac{1}{\text{N}(\Phi^{-1} (\tfrac{p}{2})|0,1)} \Bigg) \\[6pt] &= \frac{1}{2} \cdot \frac{\exp \big( -\frac{1}{2} \cdot (- \Phi^{-1} (\tfrac{p}{2}) - n (\mu - \mu_0)^2)^2 \big) + \exp \big( -\frac{1}{2} \cdot (- \Phi^{-1} (\tfrac{p}{2}) + n (\mu - \mu_0)^2)^2 \big)}{\exp \big( -\frac{1}{2} \cdot (-\Phi^{-1} (\tfrac{p}{2}))^2 \big)}. \\[6pt] \end{align}$$

As you can see, the distribution of the p-value depends on the population mean $\mu$. With some more algebra, it can be shown that the distribution of the p-value is "stochastically dominated" as $|\mu - \mu_0|$ increases (i.e., the p-value tends to get smaller in this case).

Answer 2

Let’s do two t-test examples.

In the first situation, we take $25$ observations and get a mean of $0.59218$ and variance of $1.891$. Running through the one-sample t-test calculations, we get a t-stat of $2.1532$ and a p-value of $0.04157$, significant at the legendary $0.05$-level.

In the second situation, we take $250,000$ observations and get a mean of $0.0245$ and variance of $0.9948$. These result in a t-stat of $12.283$ and a p-value of $\approx 0$.

The p-value is much smaller for the situation with the smaller observed effect!

What’s going on is that the p-value is sensitive to the sample size. This makes it do what it is supposed to in order to contradict the null hypothesis, but it does not just measure the effect size.

Answer 3

P-values are used as a measure of effect size all the time.

The simplest way to express effect sizes are

• The raw or absolute effect sizes, like a difference between means.

Some alternatives to express (relative) effect sizes are

• Cohen's D, which expresses effect size relative to the pooled population variances

• t-statistic, which expresses effect size relative to the variance of the sample means.

And also

• p-values, which are a way to express the effect size relative to the statistical probability when some null hypothesis is true.

Depending on the application one or the other may be preferred. In publications you may often see that multiple values are reported. For instance you could something like:

Our study found that participants who drank coffee had a statistically significant increase in productivity, with a mean effect size of $0.8$ ($t_{df=23}=3.2$, $p = 0.002$), proving once and for all that caffeine is the real workhorse in your cup of joe.

^{(The content of the quote is fantasy and any resemblance to real studies are purely conincidence. I had chat-gpt make that quote for me)}

Note that these relative ways of expressing effect sizes do not always coincidence. For a given absolute effect size you can have different statistics and p-values depending on the sample sizes and estimated variances. The same p-value may occur with large and small effect sizes. The same effect size may occur with large and small p-values. The p-value is not a measure of the absolute effect size.

Answer 4

If sample size is equal, t is a function of effect size, and p is a function of p. So higher effect size is associated with lower p.

I wouldn't say " lower p-value in the tests indicate higher effect". I might say larger effect sizes are associated with smaller p-values. But why would you take a measure that is fairly easy to interpret and turn it into one that is difficult to interpret (and very commonly misinterpreted).

Answer 5

The p-value is the probability that the chosen test statistic would be as large or larger than you observed from the data, given the null hypothesis.

Typically, our null hypothesis is something like "this population parameter is equal to this constant value" and our test statistic is generally chosen in a way that will provide evidence against the null hypothesis if it's false - i.e. something that, if the population parameter is not the given value, will give you results that are highly improbable under the null hypothesis. And of course many hypothesis tests are trying to detect whether there's some kind of underlying effect going on - whether it's "this coin is unfair" or "giving the COVID vaccine makes people less likely to catch the disease" or "every basketball team signs players of roughly equal height".

So, if we're performing something resembling a normal hypothesis test, a larger effect (i.e. a bigger gap between the truth and the "everything is normal" null hypothesis) will give you lower p-values (at least in expectation) if everything else remains the same.

That said, the reasons we tend to not say that p-value is not a measure of effect size are:

1. It assumes that there even is an "effect size" in the first place. If the test is comparing how people rank in some list before and after an intervention, then you check whether the order of the list changes but there's no real numeric effect that you're measuring.

2. The test statistic is a function of your observed data, i.e. your sample, which means that it's a random value, which means that it can't be a consistent measure of the effect size. As an easy example, consider the case where the null hypothesis is true - then the p-value is literally just a measure of how unusual your data happens to be by sheer chance, meaning that you will get a p-value of 0.05 or smaller roughly 1 time in 20. If you compare 20 basketball teams to the Warriors, you should expect to see at least one tiny p-value even if the teams were formed by getting ~1000 people in a room together and just drawing their names from a hat.

3. P-values are probabilities, so their relationship to effect size tends to be highly non-linear. So even if you do take all of the above into account and deal with the probabilistic components and yada yada, then turning that p-value into an actual measure of the effect size may be a massive pain.

Answer 6

A real world example. There is strong evidence (low P value) that bacon is carcinogenic. But the risk is low. There is weak evidence that smoking cannabis leaf is carcinogenic, but a priori it is likely to be as risky as smoking tobacco. The difference is that there are many bacon eaters, they don't mind admitting they eat bacon, and there is a high level of health surveillance of them. By contrast smoking weed is illegal in most countries, especially ones with good health care, so surveillance is poor.

The difference between strength of effect and strength of evidence is important.