In absence of a uniformly most powerful test, are hypothesis tests subjective?

We know that uniformly most powerful tests don’t always exist for composite alternative hypotheses. When they don’t, how can we justify the choice of test, e.g., the choice of test-statistic in a hypothesis test?

I suppose it could still be based on power considerations, i.e., we could choose tests that maximise the power for particular choices of alternative hypothesis within the composite model? Is this a subjective choice?

Are there connections to subjectivity in Bayesian statistical approaches? Should we perform sensitivity checks to check the dependence of our result on our choice of test?

• Whether "subjective choice" is the right wording is probably...subjective :-). Indeed, different test statistics will then be powerful against different alternatives. Maximization of weighted average power then is a possible route. Here is a paper in the context of unit root testing dealing with that problem: onlinelibrary.wiley.com/doi/abs/10.1111/1468-0262.00447 Commented Feb 23, 2021 at 9:36
• Thanks, can you expand into an answer why it might not be the right wording? and if not, what would be the right wording? Commented Feb 24, 2021 at 5:54
• I did not say it was not the right wording, but just that some people would not word it like that. Indeed, there is, in the absence of a UMPT, scope for directing power against different alternatives, and, given that we do not know the nature of the alternative (else, why test?), there is a user choice to be made (even weighting requires a weighting scheme) and it is fine, I would say, to call that subjective. I will see if I can put together something that is worthy of an answer. Commented Feb 24, 2021 at 8:43
• Are you familiar with the Neyman-Pearson framework? Or basically, the framework of by fixing a level $\alpha$, we find the "best" test statistic among all statistics that whose type I error is less than $\alpha$?
– Tan
Commented Mar 1, 2021 at 18:01
• @tan, yes, are you familiar with non-existence of umpt? Commented Mar 1, 2021 at 23:34

Perhaps more precisely to your question: you might wonder if a test isn't UMP, what other optimality criteria might we try to identify among a family of tests. Another characteristic to identify in a test is that of being minimax. That is the maximum risk (formally, the maximum expected loss among all possible true values of $$\theta$$) is less than among the other tests. https://link.springer.com/content/pdf/10.1007%2F0-387-27605-X_8.pdf Minimax tests guarantee power like UMP tests, and robustify analyses by incorporating a Bayesian approach to quantifying uncertainty with a loss function and a prior.