this is a question that has been lingering in my mind for a very long time, but came up again in practice and I thought I'd reach out and ask about it.
The situation I'm interested in is when you are performing a Null Hypothesis Test and you can generate multiple test statistics with your data. You know, both analytically and confirmed via simulation, the distribution of these different test statistics. When you compute p-values using your actual observed data against these distributions, you get different results. Similar results, it must be said, but different. Different enough that were one to hew to the traditional critical values to reject a null hypothesis, one might make different decisions for the different test statistics.
A little bit more concretely, consider the Likelihood Ratio Test in addition to whatever usual test statistic you are going to look at. For example, a t-statistic when comparing sample means. -2 times the natural log of the Likelihood Ratio is going to follow a Chi-Square(1) distribution, which can be confirmed via simulation or analytically. The t-statistic is obviously going to follow a t-distribution. With your actual observed data, you compute p-values using both test statistics and get different results. Which one should be used?
More concretely, I was doing homework question 8.6 in Casella and Berger. The problem setup there is X = sum of n exponential(theta) variables, and Y = sum of m exponential(mu) variables. The null hypothesis is that the rates theta and mu are the same. Sums of exponentials are gamma random variables, and the ratio X / (X + Y) of gamma random variables is a beta(n,m) random variable (I have confirmed this analytically and via simulation). So I can use a beta(n,m) to generate p-values based on observed data. I can also generate the likelihood ratio, which follows a chi-square(1) distribution, which I have confirmed via simulation. The two test statistics lead to different p-values, and they are different enough that I felt compelled to reach out and ask this community for advice.
To summarize my question, when different test statistics can be used to generate p-values, which test statistic and p-value should be chosen?