Why is it true that a sampling distribution of a test statistic is easier to derive under the null?

This is a follow-up question of the lively discussion Why is the null hypothesis often sought to be rejected?

In particular, I was wondering what @whuber meant when he referred to @StasK 's answer in the comments

Fisher and Neyman-Pearson [...] were driven primarily by this crucial technical issue to create the asymmetry between the null and alternate hypothesis."

I am not a statistician and cannot make perfect sense of what it means to say:

that a sampling distribution of the test statistic is easier to derive under the null

and b) (part of @whuber 's comment)

to create the asymmetry between the null and alternate hypothesis

So my questions are:

Q1) Why is it true that a sampling distribution of a test statistic is easier to derive under the null?

Q2) What does it mean that a test statistic is easier to derive under the null?

Q3) What does it mean to create an asymmetry between the null and alternative hypothesis? What would it mean if there was a symmetry between the null and alternative hypothesis?

I would appreciate if answers could also contain an example to illustrate this (if this is necessary at all).

• The very concept of a null hypothesis makes no sense unless it determines a definite distribution of the test statistic! Without that, it would be impossible to compute a p-value or (except in highly unusual situations) find a rejection region with a definite size. This requirement is not imposed on the alternate hypothesis (there is no p-value associated with it), which is why the two stand in an asymmetric relationship. – whuber Dec 7 '15 at 16:17
• Thanks for the comment @whuber! The concept is understood. I mostly had some difficulties understanding/interpreting/using the terms "easier derive" and "asymmetry" in this context and needed an example to fill this gap. @christoph-hanck did this nicely. – Stefan Dec 7 '15 at 16:27

Here is the easiest example I can think of to make the point.

Consider $X\sim N(\mu,1)$, i.e., sampling from a normal population with known variance 1. Then,

$$\sqrt{n}(\bar{X}_n-\mu)\sim N(0,1)$$

If the null is true, i.e., $\mu=\mu_0$, you have automatically also already derived the sampling distribution of the test statistic $t=\sqrt{n}(\bar{X}_n-\mu_0)$ under the null.

When $\mu\neq\mu_0$, write \begin{align*} t&=\sqrt{n}(\bar{X}_n-\mu_0)\\ &=\sqrt{n}(\bar{X}_n-\mu+\mu-\mu_0) \end{align*} This is the $N(0,1)$ random variable plus the deterministic quantity $\sqrt{n}(\mu-\mu_0)$, so $t\sim N(\sqrt{n}(\mu-\mu_0),1)$.

So:

Q1) Getting the distribution under the alternative was a little trickier even in this arguably very simple example.

Q2) I do not quite understand this question (or its difference to Q1) - the test statistic must be the same under H0 and H1 - in practice we do not know which of the two is true, so if the test statistic did depend on which is true, hypothesis testing would be impossible (a good thing, some would argue ;-) )

Q3) Asymmetry - I suppose (see the connect by whuber, though) - refers to that the test statistic behaves differently depending on whether H0 or H1 is true, and this is what we want and need: If the null is false, we want the test to be able to detect that. Now, if the test statistic had the same distribution ("behavior") under H0 and H1, there would be no reason to interpret a large value of the test statistic as evidence in favor of H1. As the above example demonstrates, this is also the case here: under H1, the mean of the statistic is shifted away from zero, so that the statistic is more likely to produce large realizations. Plausibly, that effect becomes stronger the larger the sample size.

• Thanks @christoph-hanck, that helps a lot! What does the abbreviation "r.v." stands for? – Stefan Dec 7 '15 at 16:30
• I just made an edit to clarify. – Christoph Hanck Dec 7 '15 at 16:33