Why does a small $p$-value indicate incompatibility with the null?

Question

Let's take, as a simple example, the two-tailed one-sample hypothesis test on the population mean. Suppose we've determined an $\alpha$-level a priori.

Let $X_1, \dots, X_n \overset{\text{iid}}{\sim}\mathcal{N}(\mu, \sigma^2)$. In this setting, given a value $\mu_0$, we have the null and alternative hypotheses $H_0: \mu = \mu_0$ and $H_1: \mu \neq \mu_0$.

Let $\bar{X}_n$ be the sample mean of $X_1, \dots, X_n$ and $S^2$ be the unbiased estimator of $\sigma^2$, with $\bar{x}_n$ and $s^2$ being the observed values.

We know that $$\dfrac{\bar{X}_n - \mu}{\sqrt{S^2/n}} \sim t_{n-1}$$ i.e., a $t$-distribution with $n-1$ degrees of freedom. Under $H_0$, we have that $$\dfrac{\bar{X}_n - \mu_0}{\sqrt{S^2/n}} \sim t_{n-1}\text{.}$$ Then we compute a $p$-value $$p = \mathbb{P}\left(|T| \geq \dfrac{\bar{x}_n - \mu_0}{\sqrt{s^2/n}} \right)$$ where $T \sim t_{n-1}$ and if $p < \alpha$, we reject $H_0$ and state there is evidence for $H_1$.

Now, I've done this procedure for years, and I'm a bit embarrassed to ask this, given that I hold a MS degree: but exactly why does having $p < \alpha$ indicate incompatibility with $H_0$ and evidence for $H_1$? Mathematically, all it is at the end of the day is the probability that your random variable $T$ takes on a value at least as extreme (in absolute value) than the one yielded by the sample. But I fail to see why having $p < \alpha$ indicates that we have evidence to reject $H_0$.

Perhaps this may have been covered in Casella and Berger and I've forgotten the details.

Answer 1

Let's use an analogy.

You wake up confused about what day it is. Worse, you don't even know the month, though you have a hunch that it might be summer, but you want it to be winter (so $H_0: \text{summer}$ and $H_a: \text{winter}$). You don't trust the calendar on your phone, but you trust the weather app, so you check it out for the temperature.

You see that the weather app reports the temperature as $-24^{\circ} C$.

You know that being that cold or colder is very unlikely during the summer, so you reject the idea that it is summer in favor of concluding that it's winter.

In this analogy, the critical value giving sufficiently small $p <\alpha$ is the temperature at which you would so doubt your hunch that it is summer that you would conclude, "Nope, winter time!"

Answer 2

I always see the p-value as an indicator of an anomaly: an unlikely extreme observation (how much unlikely, that is indicated by the p-value).

Not all discrepancies between null theory and observation are a strong indicator of incompatibility with the null. Because of noise or other variations of measurement, some discrepancy is to be expected and it is likely to get an observation within some range.

However, large discrepancies outside the probable range are unexpected. Such discrepancies are an indicator that the null theory might be incorrect. The more unexpected the discrepancy (the lower the p-value) the stronger it indicates that the null theory is incompatible with observations.

When testing a theory, by looking at a discrepancy between theory and observation, we are typically only interested in highly unlikely discrepancies.

Answer 3

Strictly speaking, any p-value is some evidence regarding the $H_0$ vs. $H_1$ question. It usually boils down to decision-making: Should you act (or plan your future acts) assuming that $H_0$ is true, or should you hold $H_1$ for true? In an empirical field you can never know with absolute certainty, but still, you have to make the decision somehow.

Now, it's a different question whether probability by itself is the right criterion for making that decision, but let us assume that it is. Then, by setting $\alpha$ to some value (usually 0.05) you are basically establishing a decision boundary: If the p-value is below it, you decide to act as if $H_1$ were true, because it is sufficiently improbable (though still possible) to get such an extreme value of $T$ if $H_0$ were right.

For example:

Assume you have ordered 1 million of 1 k$\Omega$ resistors from a manufacturer of electronic components. Due to the manufacturing process, no resistor is exactly 1 k$\Omega$, so the true resistance is some random distribution around that value. You don't have the resources to check each resistor yourself, but you can take a sample, measure the resistance on it and do the statistics.

If you get a sufficiently large p-value, $p \gt \alpha$, you can say:

Assuming that the true resistance in the population is 1 $k\Omega$, it is reasonably probable to draw a random sample whose average resistance deviates at least as much as measured from that ideal value. I'll accept the shipment and build the resistors into my product.

This is failing to reject $H_0$. On the other hand, if your p-value is below your $\alpha$, your reasoning is the following:

Assuming that the true resistance in the population is 1 $k\Omega$, it is very improbable to take a random sample whose average resistance deviates at least as much as measured from that ideal value. Hence, the true resistance likely isn't 1 $k\Omega$. I'll reject the shipment, sue the manufacturer, search for a more reliable one or whatever, but I will not use these resistors in my product, because it is not going to work properly with wrongly dimensioned components.

This is rejecting $H_0$ in favour of $H_1$.