A statistical estimator can be characterized – among others – by its bias and variance. These are different aspects, but if one is willing to accept a certain loss function (risk) than they can merged; the most popular example of which is MSE (which can be shown to be the square of the bias plus the variance). If one is willing to accept MSE as a metric, it can be used to objectively compare estimators even if they have different bias.

My question is: do we have analogous concepts for statistical tests?

Motivation: We all know that – despite being notoriously recommended, even by textbooks – it is a bad idea to select the applied test based on another test performed on the same sample. (Let's say for concreteness: to decide whether to use $t$-test or Welch-test based on an $F$-test, or Levene-test.) Doing so will result in an invalid test, i.e. the distribution of the $p$-values will no longer be uniform, the Type I error rate will be different from the significance level. However, one might say: "OK, I understand that there is some invalidity here, but, hey, we will have higher power!". (As $t$-test has higher power than Welch-test – that's of course the very reason why this flawed strategy is employed at all.)

Now, my feeling is that this invalidity in a statistical test is analogous to the bias of an estimator (actually, it is called bias in mathematical statistics texts; if I understand this concept correctly) and I have the feeling that power is analogous to variance.

So, question #1: are my feelings correct? Do we really have an equivalence here?

Question #2: if so, do we have an analog for MSE? (I accept that we have to select a loss function (risk) for this of course.)

This would be interesting because it would allow us to objectively decide whether there is any merit in the above reasoning (i.e. higher power offsetting the loss of validity – just as a biased estimator can have smaller MSE than an unbiased one).

  • $\begingroup$ Why isn't this just an issue of p-value adjustment for multiple testing? $\endgroup$ – Michael Chernick Mar 29 '18 at 2:13
  • 1
    $\begingroup$ @MichaelChernick This "preliminary testing" strategy implies a totally different mechanism (one that is much harder to track analytically, actually) for the change in Type I error rate than multiple comparisons. See this article,but it has been also discussed here, see this or this in addition to the one I linked. $\endgroup$ – Tamas Ferenci Mar 29 '18 at 4:52
  • 1
    $\begingroup$ General comment: I don't think it's good to use any method that just moves type II error into type I error, i.e., does not work as advertised with respect to type I error. $\endgroup$ – Frank Harrell Apr 4 '18 at 13:15
  • $\begingroup$ @FrankHarrell I personally agree with you. Nevertheless, I also see some point in the reasoning "OK, the actual alpha is 5.01% instead of the nominal 5% when the null is true, but hey, when its not true, we have 20% more power!". (Of course I made up the numbers.) How should I respond to this? Instead of the "I don't think it's good" and "I personally agree" kind of approaches, an objective, quantitative answer would be of course better. And MSE achieves just this very aim (for an estimator), so that's why I was wondering whether it is possibly to do something like that for a test! $\endgroup$ – Tamas Ferenci Apr 4 '18 at 13:31
  • 1
    $\begingroup$ In the two examples I've seen analyzed in detail the gain in power was exactly equal to the gain in type I error. So unless the procedure is truly shown to only increase alpha to 0.051 when you think it's 0.05 I remain skeptical. $\endgroup$ – Frank Harrell Apr 4 '18 at 20:02

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.