How to define a Rejection Region when there's no UMP?

Consider the linear regression model

$\mathbf{y}=\mathbf{X\beta}+\mathbf{u}$,

$\mathbf{u}\sim N(\mathbf{0},\sigma^2\mathbf{I})$,

$E(\mathbf{u}\mid\mathbf{X})=\mathbf{0}$.

Let $H_0: \sigma_0^2=\sigma^2$ vs $H_1: \sigma_0^2\neq\sigma^2$.

We can deduce that $\frac{\mathbf{y}^T\mathbf{M_X}\mathbf{y}}{\sigma^2}\sim \chi^2(n-k)$, where $dim(\mathbf{X})=n\times k$. And $\mathbf{M_X}$ is the typical notation for the annihilator matrix, $\mathbf{M_X}\mathbf{y}=\hat{\mathbf{y}}$ , where $\hat{\mathbf{y}}$ is the dependent variable $\mathbf{y}$ regressed on $\mathbf{X}$.

The book I'm reading states the following: I've previously asked what criteria should be used to define a Rejection Region(RR), see the answers to this question, and the main one was to choose the RR that made the test as powerful as possible.

In this case, with the alternative being a bilateral composite hypothesis there's usually no UMP test. Also, by the answer given in the book, the authors don't show if they did a study of the power of their RR. Nevertheless, they chose a two-tailed RR. Why is that, since the hypothesis do not 'unilaterally' determine the RR?

Edit: This image is in the solution manual of this book as the solution to exercise 4.14.

• Please add a reference to the book. Related: P-value in a two-tail test with asymmetric null distribution. Jul 13 '16 at 16:05
• @Scortchi thanks for the link. Could I ask you something about this question? Do you find it interesting? I'm trying to assess whether I'm making interesting questions, or if I should direct my interests towards other areas... Jul 14 '16 at 13:35
• Not everyone finds theory interesting of course, but some people do (including me) & we've nearly 2k qs tagged with mathematical-statistics. So, a fine q. IMO. It is a little broad but I think a good answer would survey various approaches & considerations, & a motivating example helps a lot. (I'd've chosen as simple an example as possible though - tests about the variance of a normal distribution with known mean, or the mean of an exponential distribution.) [BTW I often forget to vote on qs when I comment on them.] Jul 14 '16 at 15:31
• @Scortchi thanks for your feedback. Sometimes I'm not sure if I structure well the question, since I'm self studying this. Jul 14 '16 at 16:10
• You should define $M_X$ Jul 16 '16 at 21:20

Easier to first work through the case where the regression coefficients are known & the null hypothesis therefore simple. Then the sufficient statistic is $T=\sum z^2$, where $z$ is the residual; its distribution under the null is also a chi-squared scaled by $\sigma^2_0$ & with degrees of freedom equal to the sample size $n$.

Write down the ratio of the likelihoods under $\sigma=\sigma_1$ & $\sigma=\sigma_2$ & confirm that it's an increasing function of $T$ for any $\sigma_2 > \sigma_1$:

The log likelihood ratio function is $$\ell(\sigma_2;T,n)-\ell(\sigma_1;T,n)=\frac{n}{2} \cdot \left[\log \left(\frac{\sigma_1^2}{\sigma_2^2}\right) + \frac{T}{n} \cdot \left(\frac{1}{\sigma_1^2} - \frac{1}{\sigma_2^2}\right) \right]$$, & directly proportional to $T$ with positive gradient when $\sigma_2>\sigma_1$.

So by the Karlin–Rubin theorem each of the one-tailed tests $H_0:\sigma=\sigma_0$ vs $H_\mathrm{A}:\sigma < \sigma_0$ & $H_0:\sigma = \sigma_0$ vs $H_\mathrm{A}:\sigma < \sigma_0$ is uniformly most powerful. Clearly there's no UMP test of $H_0:\sigma = \sigma_0$ vs $H_\mathrm{A}:\sigma \neq \sigma_0$. As discussed here, carrying out both one-tailed tests & applying a multiple-comparisons correction leads to the commonly used test with equally sized rejection regions in both tails, & it's quite reasonable when you're going to claim either that $\sigma>\sigma_0$ or that $\sigma<\sigma_0$ when you reject the null.

Next find the ratio of the likelihoods under $\sigma=\hat\sigma$, the maximum-likelihood estimate of $\sigma$, & $\sigma=\sigma_0$:

As $\hat\sigma^2=\frac{T}{n}$, the log likelihood ratio test statistic is $$\ell(\hat\sigma;T,n)-\ell(\sigma_0;T,n)=\frac{n}{2} \cdot \left[\log \left(\frac{n\sigma_0^2}{T}\right) + \frac{T}{n\sigma_0^2} - 1 \right]$$

This is a fine statistic for quantifying how much the data support $H_\mathrm{A}:\sigma \neq \sigma_0$ over $H_0:\sigma = \sigma_0$. And confidence intervals formed from inverting the likelihood-ratio test have the appealing property that all parameter values inside the interval have higher likelihood than those outside. The asymptotic distribution of twice the log-likelihood ratio is well known, but for an exact test, you needn't try to work out its distribution—just use the tail probabilities of the corresponding values of $T$ in each tail.

If you can't have a uniformly most powerful test, you might want one that's most powerful against the alternatives closest to the null. Find the derivative of the log-likelihood function with respect to $\sigma$—the score function:

$$\frac{\mathrm{d}\,\ell(\sigma;T,n)}{\mathrm{d}\,\sigma}=\frac{T}{\sigma^3} - \frac{n}{\sigma}$$

Evaluating its magnitude at $\sigma_0$ gives a locally most powerful test of $H_0:\sigma=\sigma_0$ vs $H_\mathrm{A}:\sigma \neq \sigma_0$. Because the test statistic's bounded below, with small samples the rejection region may be confined to the upper tail. Again, the asymptotic distribution of the squared score is well known, but you can get an exact test in the same way as for the LRT.

Another approach is to restrict your attention to unbiased tests, viz those for which the power under any alternative exceeds the size. Check your sufficient statistic has a distribution in the exponential family; then for a size $\alpha$ test, $\phi(T)= 1$ if $T<c_1$ or $T>c_2$, else $\phi(T)= 0$, you can find the uniformly most powerful unbiased test by solving \begin{align} \operatorname{E}(\phi(T)) &= \alpha \\ \operatorname{E}(T\phi(T)) &= \alpha \operatorname{E} T \end{align}

A plot helps show the bias in the equal-tail-areas test & how it arises: At values of $\sigma$ a little over $\sigma_0$ the increased probability of the test statistics' falling in the the upper-tail rejection rejection doesn't compensate for the reduced probability of its falling in the lower-tail rejection region & the power of the test drops below its size.

Being unbiased is good; but it's not self-evident that having a power slightly lower than the size over a small region of the parameter space within the alternative is so bad as to rule out a test altogether.

Two of the above two-tailed tests coincide (for this case, not in general):

The LRT is UMP among unbiased tests. In cases where this isn't true the LRT may still be asymptotically unbiased.

I think all, even the one-tailed tests, are admissible, i.e. there's no test more powerful or as powerful under all alternatives—you can make the test more powerful against alternatives in one direction only by making it less powerful against alternatives in the other direction. As the sample size increases, the chi-squared distribution becomes more & more symmetric, & all the two-tailed tests will end up being much the same (another reason for using the easy equal-tailed test).

With the composite null hypothesis, the arguments become a little more complicated, but I think you can get practically the same results, mutatis mutandis. Note that one but not the other of the one-tailed tests is UMP!

• Scortchi thanks for your answer. I still have some doubts, though. Firstly, could you elaborate a bit more on the following sentence? «applying a multiple-comparisons correction leads to the commonly used test with equally sized rejection regions in both tails, & it's quite reasonable when you're going to claim either that σ>σ0 or that σ<σ0 when you reject the null.» Also why do you say it's reasonable? I think this is the core of my question if I'm not mistaken. ;) Jul 18 '16 at 17:52
• I read this paragraph from you linked answer, but I did not understand it well«Doubling the lowest one-tailed p-value can be seen as a multiple-comparisons correction for carrying out two one-tailed tests.» I would be thankful if you could please explained it a bit more. ;) Jul 18 '16 at 19:14
• See Bonferroni correction. If you carry out two separate size $\alpha/2$ tests the family-wise Type I error is no more than $\alpha$, & when the rejection regions are disjoint it's exactly $\alpha$. I wanted to point out that the equal-tail-areas test can be seen in this way because people sometimes seem to think the only reasons to use it are ease of calculation & approximation to the other tests. In fact each test has its own rationale: so I wouldn't say this was the core of your question; it's a matter of horses for courses. Jul 19 '16 at 11:39

In this case, with the alternative being a bilateral composite hypothesis there's usually no UMP test.

I am not sure if that is true in general. Certainly, a lot of the classical results (Neymon-Pearson, Karlin-Rubin) are based on either simple or one-sided hypothesis, but generalizations to two-sided composite hypothesis do exist. You can find some notes on that here, and more discussion in the textbook here.

For your problem specifically, I don't know whether a UMP test exists or not. But intuitively, it seems to be that under 0-1 loss, a one sided test will probably be inadmissible, and thus the class of admissible test will be all two-sided tests. Give the class of two sided tests, the goal is to find the one with the largest power, which should automatically happen by choosing quantiles around the one mode of the $\chi^2$. (This is all based on intuition).

• There's clearly not a uniformly most powerful test in this case because of the existence of different tests most powerful against particular alternatives in different directions from $\sigma_0$. For a "best" test defined in terms of power you'd have to look for the uniformly most powerful test of all unbiased tests, or of all invariant tests; or for a locally most powerful test; or something like that - & perhaps end up settling for any admissible test. Jul 16 '16 at 12:50