Relationship of significance level and p-values

Question

In a hypothesis test, the null hypothesis $H_0$ is rejected in favor of $H_1$ if $|\bar X|>\mu$.

When tests found that $\bar x$ is the mean of the samples, a p-value is computed using

$$ p=\Pr(|\bar X|>\bar x) $$ Using (e.g $\bar X\sim t(n-1) $ the p-value is obtained and the conclusion is:

Tests with significance level $\alpha≥p$ rejects the null hypothesis

---

This does not make sense to me. What does the p-value have to do with the significance level?

$p$ only tells the probability of obtaining $\bar X ≥\bar x$ or values as extreme as $\bar x$

On the other hand, $\alpha=\Pr(\bar X\in C)$ where $C$ is the critical region (i.e. $\Pr(|\bar X|>\mu)$) or the probability of exceeding the value $\mu$ as set by the practicioner.

I am struggling to see any formal connection between the two. Please help me connect their concepts. Thank you

Answer 1

This is more of a commentary in a broader sense but worth mentioning.

As framed in $\mathrm{ [I]}$ (sec. $14.2.4,$ p. $688$), $p$-value was developed by Fisher to (emphasis mine)

[...] evaluate the worst-case scenario for the null hypothesis in the sense that it involves the observed value of the statistic and realizations more damning for the null. It measures the probability of observing a sample realization that would produce a statistic value equal to or worse than the one already observed.
[I]f the p-value is small this implies that either the observed realization of the test statistic constitutes a very rare event or the postulated null hypothesis is invalid. In cases where the p-value is small the first choice is considered practically impossible and the modeler adopts the view that the postulated hypothesis is invalid. ... In the early stages of his work Fisher suggested $p$-values of $0.05$ and $0.01$ to be used as intuitive thresholds. Later on, however, he insisted that one should separate the $p$-value from the decision to accept or reject $H_0.$

The author conspicuously reminds (sec. $14.2.5,$ p. $692$) the readers:

In a certain sense the $p$-value represents the worst-case scenario for the null hypothesis, taking into account not just the observed sample realization but also more unfavorable realizations. The question of accepting or rejecting $H_0$ is a separate issue and the $p$-value should not be confused with the significance level of the Neyman–Pearson testing framework [...]

and again in the same vein (sec. $14.3.5,$ p. $703$):

Even though the two can be related mathematically, they constitute very different notions in different contexts. The role of the $p$-value is inferential and that of the significance level is decision making. This is why the use of $p$-values in the context of the Neyman–Pearson approach was considered to be a sacrilege by Fisher ($1956$).

Please check $p$-value: Fisherian vs. contemporary frequentist definitions and other links therein.

---

Reference:

$\rm [I]$ Probability Theory and Statistical Inference: Econometric Modeling with Observational Data, Aris Spanos, Cambridge University Press, $1999.$

Answer 2

I assume that $\mu$ is the critical value at test level $\alpha$. Then the test rejects the $H_0$ if observed $|\bar x|\ge \mu$ (the case of equality is irrelevant if the distribution of $\bar X$ is continuous, but "$\ge$" here will correspond to "$\ge$" in "$\alpha\ge p$"). Also, ${\rm Pr}\{|\bar X|\ge\mu\}={\rm Pr}(\bar X\in C)=\alpha$ (assuming that the test is exact).

Note that the t-test relies on a test statistic that has $\bar X$ linearly transformed and involves the standard error; the argument there is the same (using the statistic $T$ instead of $\bar X$) but I will here focus on just $\bar X$ as test statistic as in the question, which is the case in a Gauss-test with known variance and the null hypothesis that the underlying mean is zero. Note also that $\mu$ is more often used to denote the hypothesised mean rather than the critical value, but I'll stick to the notation in the question.

Now $p={\rm Pr}\{|\bar X|\ge |\bar x|\}$ (note that the absolute value should here also be taken of the $\bar x$). If $\alpha\ge p$, this means that $|\bar x|\ge \mu$, and therefore the test rejects, because it means that the set $\{|\bar X|\ge |\bar x|\}$ must be a subset of or equal to the set $\{|\bar X|\ge \mu\}$. Note that it can only be a subset, equal, or a superset, and it can't be a superset, because then $\alpha<p$ (assuming $\mu$ to be chosen as the smallest possible value that makes ${\rm Pr}\{|\bar X|\ge\mu\}=\alpha$, which as appropriately defined critical value it should be - if the density of $\bar X$ is larger than zero around $\mu$ as in the Gauss-test, this is automatically the case).

In many tests including the Gauss-test the probability that $p=\alpha$ exactly is zero, and ignoring this possibility would make this explanation slightly less tedious.