But how is this a useful value? Even if the p-value is low, couldn't we have a similar, or even lower likelihood of observing an extreme effect if the null hypothesis is false?

Yes, this is certainly possible.

### p-values for tests without power

• It happens clearly when we would define a very silly hypothesis test.

For a normal distributed variable $$X \sim N(\mu,1)$$ we can define a test $$H_0:\mu = 1$$ which is 'reject the null hypothesis when some observation follows $$-0.627 \leq X_{obs} \leq 0.627$$'. With this rule, the probability for a type I error (probability rejecting the null hypothesis when the null hypothesis is actually true) will be 5%.

• So p-values do not say everything and we should also aim for tests with high power.

High power means that the probability to reject the null hypothesis is high if is some given alternative hypothesis is true. You get the highest power when you choose the rejection region where the observations have a higher probability under the alternative hypothesis (or multiple hypotheses) than under the null hypothesis. (this is the likelihood-ratio test)

While the example above is a silly test (just to make the principle very clear) it can also happen to some extent in serious tests (and p-values are not always so easy to define, just think about something simple as comparing two means). However, in practice, this will not happen because an experiment will be designed to have high power in relation to some alternative hypothesis. For most experiments, the observed statistic is also equated with some effect size via maximum likelihood (so whatever effect we measure, it will be having a larger likelihood than the null-'no effect'-hypothesis).

### What is extreme?

Some common interpretation is that p-values indicate how extreme a certain value is in terms of the probability that this 'level of extremeness' would occur.

A problem with these p-values that relate to probabilities of anomalies or probabilities of extreme values, is that it is not always clear what is to be considered extreme, merely by looking at the probability of an event. The situation can be turned completely upside down.

If you want to test whether a coin is fair then you could flip it a lot of times and see whether the ratio of heads and tails is a lot far away from 1:1, but what do you do when it is suspiciously close to 1:1? What is extreme depends on what alternative you have in mind. And both cases may occur.

• Probably the first use of p-values was in 1710 by John Arbuthnot in An argument for divine providence, taken from the constant regularity observed in the births of both sexes. He was analyzing the yearly birth ratio's of Londen boys:girls. The ratio's are remarkably different from the middle, and for 82 years there had been more boys than girls. The probability to get boys>girls 82 times in a row is very small $$\frac{1}{2}^{82} \sim \frac{1}{4 \,8360\,0000\,0000\,0000\,0000\,0000}$$ he ends up at the conclusion that it is very unlikely that boys and girls are born in the same ratio (and as alternative this must be some (divine) providence to counter the greater mortality among men to finally end up with equal men and women).

• An example where a small deviation is considered as extreme is this question: A chart of daily cases of COVID-19 in a Russian region looks suspiciously level to me - is this so from the statistics viewpoint?

Say, the model is that the data is Poisson or Binomial distributed and the null hypothesis is that the mean $$\mu = 100$$. Then, if you observe every day that the values are very close 100, then this is not extremely different from the null hypothesis. However, instead of doubts about the null hypothesis, one may now start to doubt the model (the data does not follow a random Poisson distribution).

Similar is Fisher's Mendelian paradox, which relates to Fisher's finding that some ratio's of phenotypes in Mendel's field experiments had a consistently very small discrepancy from the expected mean.

### Interpretation as test accuracy

We can have two interpretations of p-values.

• P-values, although a Frequentist concept, are interpreted in some sort of a Bayesian manner. With this interpretation, the p-values change our degree of belief in the null hypothesis. An example is the prosecutor's fallacy. Some events might be very unlikely and extreme but that does not mean that the null hypothesis is to be rejected (because the prior belief is that the alternative probability is very unlikely). In this way, people can sometimes be a bit variable/flexible with the use of p-values. There is no single standard and the use of cut-off values 0.01 or 0.05 are rules of thumb and can change depending on the situation (e.g. the Sagan standard).

• P-values are a measure for the accuracy of the measurement. We use p-values in experimental settings where theories are being tested and the null-hypothesis (null referring to 'no effect') is being challenged.

This challenge can be of the theory 'an apple and a feather fall at the same speed' and it can be tested by repeatedly letting the apple and the feather fall and record which one touches the floor first. I could do this with 8 repetitions and reject the null hypothesis (they fall with the same speed and therefore are equally likely to touch the floor first) when either the feather touches the floor 8 times or the apple touches the floor first 8 times. But probably I might already draw my conclusions after one single repetition.

Why then do we need p-values? We need them for the cases when the experiment is subject to variability. Our measurements are not just clear examples as comparing whether an apple or feather falls faster than the other. Often, measurements have noise and random behavior is underlying observations. In such cases, we need to be able to express how likely it is that a certain effect is being observed given the null (no effect) hypothesis.

In these settings, the determined effect/theory corresponds to the maximum likelihood estimate. So in this setting, there is no situation that "couldn't we have a similar or even lower likelihood..." because the effect is the effect/theory for which the likelihood is maximized. The reason that we use the p-value is to verify how precise our measurement/experiment is, by expressing how likely a measurement of this effect size could have occurred by pure chance.