Why is it true that a sampling distribution of a test statistic is easier to derive under the null?

Question

This is a follow-up question of the lively discussion Why is the null hypothesis often sought to be rejected?

In particular, I was wondering what @whuber meant when he referred to @StasK 's answer in the comments

Fisher and Neyman-Pearson [...] were driven primarily by this crucial technical issue to create the asymmetry between the null and alternate hypothesis."

I am not a statistician and cannot make perfect sense of what it means to say:

a) (part of @StasK 's answer)

that a sampling distribution of the test statistic is easier to derive under the null

and b) (part of @whuber 's comment)

to create the asymmetry between the null and alternate hypothesis

So my questions are:

Q1) Why is it true that a sampling distribution of a test statistic is easier to derive under the null?

Q2) What does it mean that a test statistic is easier to derive under the null?

Q3) What does it mean to create an asymmetry between the null and alternative hypothesis? What would it mean if there was a symmetry between the null and alternative hypothesis?

I would appreciate if answers could also contain an example to illustrate this (if this is necessary at all).

Answer 1

Here is the easiest example I can think of to make the point.

Consider $X\sim N(\mu,1)$, i.e., sampling from a normal population with known variance 1. Then,

$$\sqrt{n}(\bar{X}_n-\mu)\sim N(0,1)$$

If the null is true, i.e., $\mu=\mu_0$, you have automatically also already derived the sampling distribution of the test statistic $t=\sqrt{n}(\bar{X}_n-\mu_0)$ under the null.

When $\mu\neq\mu_0$, write \begin{align*} t&=\sqrt{n}(\bar{X}_n-\mu_0)\\ &=\sqrt{n}(\bar{X}_n-\mu+\mu-\mu_0) \end{align*} This is the $N(0,1)$ random variable plus the deterministic quantity $\sqrt{n}(\mu-\mu_0)$, so $t\sim N(\sqrt{n}(\mu-\mu_0),1)$.

So:

Q1) Getting the distribution under the alternative was a little trickier even in this arguably very simple example.

Q2) I do not quite understand this question (or its difference to Q1) - the test statistic must be the same under H0 and H1 - in practice we do not know which of the two is true, so if the test statistic did depend on which is true, hypothesis testing would be impossible (a good thing, some would argue ;-) )

Q3) Asymmetry - I suppose (see the connect by whuber, though) - refers to that the test statistic behaves differently depending on whether H0 or H1 is true, and this is what we want and need: If the null is false, we want the test to be able to detect that. Now, if the test statistic had the same distribution ("behavior") under H0 and H1, there would be no reason to interpret a large value of the test statistic as evidence in favor of H1. As the above example demonstrates, this is also the case here: under H1, the mean of the statistic is shifted away from zero, so that the statistic is more likely to produce large realizations. Plausibly, that effect becomes stronger the larger the sample size.