Correct me if I'm wrong, but from my understanding, the standard procedure of testing whether data from an unknown source have a specific mean is to (a) perform a normality test to see if the data are normal (b) if it is, perform a $t$-test.

Now, say the normality test is passed, and let's pretend we can assume normality hereafter (because, say, we know the test has high power). The problem is, in order to perform a $t$-test, one must know the distribution of data is normal before conducting experiments and gathering data. This is because the experiment needs to be completely specified before it is executed. However, in this case, we are designing it progressively by using the conclusion drawn from the normality test to design the $t$-test.

I am not sure if progressively-designed tests are always bad, but here is another example to illustrate my concern. Let's say you know from domain knowledge that the mean of $X$ is within the range $[0, 1]$. You first test the null $E(X) > \frac{1}{2}$. Next, you test one of $E(X) > \frac{1}{4}$ and $E(X) > \frac{3}{4}$ based on the outcome of the first test, and so forth. In the end, this progressively-designed test would be equivalent to testing $E(X)=\bar{X}$, which is downright stupid because you'll always get $p=0$.

We can tell from the example above that progressively-designed tests can be deceptive in some cases. What's the rationale behind a normality test followed by a $t$-test, then? Why is it OK to design the experiment while you are executing it?

  • 6
    $\begingroup$ Regarding your first paragraph: I encourage you to look at the references given in this post at datamethods.org. There is a section called "Use of normality tests before t tests" with many pointers to publications. In summary, these sequential test procedures (i.e. first a normality test, then either $t$-test or a nonparametric test, depending on result) are not encouraged because they alter the characteristics of the tests (and do not answer the right questions). $\endgroup$ – COOLSerdash Jan 18 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.