# What are desirable characteristics of a test statistic?

I've seen definitions of statistics which combine multiple terms in a very specific way. What are advantages of these expression and why not use just any calculation on the data? For example why do you use specific forms like for the t-test, F-test, Jarque-Bera, ...?

Does it have to be independent of some data properties? Are some statistics more powerful?

For a test-statistic to be a statistical test you need to know the sampling distribution of that statistic if the null hypothesis is true. For some statistics it is easier to derive (asymptotically) what that distribution would be, and these statistics have been given names like t-statistic, F-statistic, etc. There exist many different test statistics because many will just test different null-hypotheses. Sometimes the difference is huge, sometimes the difference is extremely subtle. Sometimes different test statistics test exactly the same hypothesis. In those cases the difference could be statistical power, and sometimes it turns out to be the same test developed within different sub-disciplines of statistics and given different names.

• Because both historically and theoretically it is not the case that statistics are developed with the sole aim of obtaining something whose sampling distribution can easily be derived mathematically, there must be more to the story than this. And indeed there is: there are many criteria used to develop statistics, including invariance, most powerful (MP), unbiased, robust, and more.
– whuber
Commented Oct 31, 2014 at 14:27

tl;dr: your test needs to have statistical power, the concept of statistical power invalidates the whole raison d'etre of null hypothesis testing

Suppose you have run an experiment in a lab and you'd like to test is an effect. Anathema! Don't you know you can only reject the absence of an effect? Haven't you read Popper? Don't you know it's unscientific to try and confirm anything, you can only try to disprove things.

Very well you say, so you conceive of a null hypothesis, you construct a statistic around it, and you test how likely you are to get such an extreme value under the null hypothesis. However, you find the process tedious, and you wonder if you could automate it a little more.

You then have a brilliant idea. To reject the null hypothesis, you will write a report on your experiment (containing the data). You will then put that report through a cryptographic hash function, like sha2. Since this hash is very unpredictable, if there is no effect, the first 6 bits will only be 0 about 1/128th of the time.

Therefore, you now have a universal null hypothesis test. Hash your paper, and test if the first 6 bits are 0. If they aren't, then you can reject the null hypothesis at p=0.78%. This suggests there may be an effect, where somehow the slime mold you've been studying is inverting the hash function.

Of course, in reality, you have about a 0.78% chance of rejecting the null-hypothesis, and all your rejections will be flukes. It is said that your test has no power.

Very well then, let us use tests which have power! So how do we go about that? Well, we need an idea of what would happen if there were an effect, let's see... Horror! Despair! It seems that we must actually model our actual hypothesis... but... that is unspeakably unscientific. Hume showed that induction was impossible!

There's the dirty secret. The entire concept of null hypothesis testing is a contortion used to keep the pretense that we're not making assumptions about the effect we are after, that we are merely rejecting hypotheses. Bullshit. The effect is implicitly modeled the minute we start caring about the power of the test.

From more of an intro stat perspective, some tests are useful with some data and others are not. For example, if your data is normally distributed (or if you're looking at the means of samples greater than 30 in size) you can use a Z-test. They're easy to compute.

If you're forced to use a small sample, a better distribution to rely on is the T distribution, which takes into account the sample size. But once the sample size gets large enough, the T distribution starts to look a like Z (the normal curve).

You can construct a 95% CI and then calculate T and find that they agree: the mean is in the confidence interval--T isn't large enough to reject. It's not all toeMAYtoe toeMAHtoe but especially at the intro stat level, it's all about the amount of data you have, the distribution of the data, and what you're asking.