4
$\begingroup$

We know that uniformly most powerful tests don’t always exist for composite alternative hypotheses. When they don’t, how can we justify the choice of test, e.g., the choice of test-statistic in a hypothesis test?

I suppose it could still be based on power considerations, i.e., we could choose tests that maximise the power for particular choices of alternative hypothesis within the composite model? Is this a subjective choice?

Are there connections to subjectivity in Bayesian statistical approaches? Should we perform sensitivity checks to check the dependence of our result on our choice of test?

$\endgroup$
5
  • $\begingroup$ Whether "subjective choice" is the right wording is probably...subjective :-). Indeed, different test statistics will then be powerful against different alternatives. Maximization of weighted average power then is a possible route. Here is a paper in the context of unit root testing dealing with that problem: onlinelibrary.wiley.com/doi/abs/10.1111/1468-0262.00447 $\endgroup$ Commented Feb 23, 2021 at 9:36
  • $\begingroup$ Thanks, can you expand into an answer why it might not be the right wording? and if not, what would be the right wording? $\endgroup$
    – innisfree
    Commented Feb 24, 2021 at 5:54
  • 2
    $\begingroup$ I did not say it was not the right wording, but just that some people would not word it like that. Indeed, there is, in the absence of a UMPT, scope for directing power against different alternatives, and, given that we do not know the nature of the alternative (else, why test?), there is a user choice to be made (even weighting requires a weighting scheme) and it is fine, I would say, to call that subjective. I will see if I can put together something that is worthy of an answer. $\endgroup$ Commented Feb 24, 2021 at 8:43
  • $\begingroup$ Are you familiar with the Neyman-Pearson framework? Or basically, the framework of by fixing a level $\alpha$, we find the "best" test statistic among all statistics that whose type I error is less than $\alpha$? $\endgroup$
    – Tan
    Commented Mar 1, 2021 at 18:01
  • 1
    $\begingroup$ @tan, yes, are you familiar with non-existence of umpt? $\endgroup$
    – innisfree
    Commented Mar 1, 2021 at 23:34

1 Answer 1

3
+100
$\begingroup$

In practice, we rarely justify selection of a test due to its being uniformly most powerful (against alternatives). In fact, in most standard software and published literature, analyses of GLMs and OLS models most often defer to using the Wald test rather than the LRT. Admittedly those methods are often very close, but nominally, it is not the UMP test that is used, but the Wald. The Wald is formally preferred because of its alignment with confidence intervals. You could go so far as to argue that, if a significant LR test is rendered insignificant when performed with Wald, then to be conservative one should report the Wald results. Theory and practice diverge.

When planning analyses we often justify a specific analysis because it achieves a target power for a given sample size and alpha-level at a pre-specified effect size. It goes without saying that the test may have decent power in a neighborhood of possible effect sizes. There are however, certain unknowns that are hard to quantify and harder to anticipate, such as contamination (the tendency of study participants to exchange knowledge of the study and treatments among each other), imbalanced clusters (family studies with differing levels of attrition over time), accidental unblinding (administration of rescue therapy, emergency unblinding issues) where in retrospect one might have preferred another testing modality to the one that was proposed.

Perhaps more precisely to your question: you might wonder if a test isn't UMP, what other optimality criteria might we try to identify among a family of tests. Another characteristic to identify in a test is that of being minimax. That is the maximum risk (formally, the maximum expected loss among all possible true values of $\theta$) is less than among the other tests. https://link.springer.com/content/pdf/10.1007%2F0-387-27605-X_8.pdf Minimax tests guarantee power like UMP tests, and robustify analyses by incorporating a Bayesian approach to quantifying uncertainty with a loss function and a prior.

$\endgroup$
2
  • 1
    $\begingroup$ Thanks. Who is we? Which community are you speaking for? $\endgroup$
    – innisfree
    Commented Mar 1, 2021 at 23:37
  • $\begingroup$ Speaking as a professional statistician. $\endgroup$
    – AdamO
    Commented Mar 1, 2021 at 23:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.