In general, the process of hypothesis testing can be divided in 4 steps:
- Formulate the practical problem in terms of hypotheses.
- Calculate a statistic $T$, a function purely of the data. All good test statistics should have two properties: (a) they should tend to behave differently when $H_0$ is true from when $H_1$ is true; and (b) their probability distribution should be calculable under the assumption that $H_0$ is true.
- Choose a critical region. We must be able to decide on the kind of values of $T$ which will most strongly point to $H_1$ being true rather than $H_0$ being true.
- Decide the size of the critical region. This involves specifying how great a risk we are prepared to run of coming to an incorrect conclusion. We define the significance level or size of the test, which we denote by $\alpha$, as the risk we are prepared to take in rejecting $H_0$ when it is in fact true.
It seems the most creative step, the one that really sets a specific test apart from others is the choice of the statistic $T$. Hence, my question is: How did authors of statistical hypothesis tests come up with their statistics?
Given a specific problem, is it always obvious what the ideal (if this is definable on objective grounds at all) statistic ought to be? It seems those two requirements listed in step 2 above are two broad and many different statistics could be devised to test the same hypotheses. For instance, would it have not been a different alternative test to the t-test based on medians or other statistic...?