Invariance Principle of MLE With Regard to Partitioned Parameter

Question

Some days ago I asked a question here on CV regarding the invariance principle of MLE. By now I could figure out how this principle works (and that is in fact pretty easy to show). The MLE $\hat\theta$ can be partitioned into, say, 2 partitions, i.e. $\hat\theta = (\hat\theta_1,\hat\theta_2)$ and it seems to be true that one can obtain $\hat\theta$ by these two steps:

• Step 1: $$\hat\theta_1(\theta_2) = \operatorname*{argsup}_{\theta_1}L\Big(\theta_1,\theta_2\Big)$$
• Step 2: $$\hat\theta_2 = \operatorname*{argsup}_{\theta_2}L\Big(\hat\theta_1(\theta_2),\theta_2\Big)$$

Then the MLE is given by $\hat\theta = \Big(\hat\theta_1(\hat\theta_2),\hat\theta_2\Big)$.

This pretty much looks like some invariance property argument and I think I can state it verbally: Step 1 yields the MLE of of the true value of $\theta_1$ because if $\hat\theta$ is the MLE then $\text{proj}_1(\hat\theta) = \hat\theta_1$ is the MLE of $\theta_1$ by the invariance principle. Since $\hat\theta_1(\theta_2)$ is a function of $\theta_2$, the same argument applies. That is, $\hat\theta_1(\hat\theta_2)$ is the MLE of $\hat\theta_1(\theta_2)$. But why does the MLE derived from the above steps coincide with $\hat\theta$? Is this result really a corollary from the invariance principle or is it a different theorem?

Answer 1

You question relates to the principle of the iterated supremum, which (loosely stated) says:

$$\sup_{\theta_1, \theta_2} L(\theta_1, \theta_2) = \sup_{\theta_2} \Big( \sup_{\theta_1} L(\theta_1, \theta_2) \Big).$$

This theorem says that you can find the supremum of a multivariate real function in an iterative manner, by taking the supremum one argument at a time. (Note that each time you take the supremum over an argument parameter, the result is a function of any remaining parameters.)

Now, if each of the supremum operations in your iteration has an argument value that gives the maximum$^\dagger$ (i.e., the $\text{argmax}$ exists at each step, yielding the resulting supremum), then you can obtain an analogous result for the $\text{argmax}$, and so you can then obtain the MLE via iteration. Of course, once you want to replace $\sup$ with $\text{argmax}$ things get a bit trickier, because the whole idea of the supremum is that it is well-defined (for real functions) even when there is not a maximising value. So if you want to formulate an analogous iterative property for the maximising argument, you need to restrict attention to functions that have well-defined maxima, which means that you are going to get a less general principle. Roughly speaking, you will find that this iterative method works to find the MLE in a wide class of cases (but not all of them).

---

$^\dagger$ I note that in your question you use the term $\text{argsup}$, which is really unclear (and whuber is right to question this in the comments). If there is an argument that attains the supremum then this is an $\text{argmax}$ in the usual sense; calling this $\text{argsup}$ just leads to confusion.