1
$\begingroup$

Some days ago I asked a question here on CV regarding the invariance principle of MLE. By now I could figure out how this principle works (and that is in fact pretty easy to show). The MLE $\hat\theta$ can be partitioned into, say, 2 partitions, i.e. $\hat\theta = (\hat\theta_1,\hat\theta_2)$ and it seems to be true that one can obtain $\hat\theta$ by these two steps:

  • Step 1: $$\hat\theta_1(\theta_2) = \operatorname*{argsup}_{\theta_1}L\Big(\theta_1,\theta_2\Big)$$
  • Step 2: $$\hat\theta_2 = \operatorname*{argsup}_{\theta_2}L\Big(\hat\theta_1(\theta_2),\theta_2\Big)$$

Then the MLE is given by $\hat\theta = \Big(\hat\theta_1(\hat\theta_2),\hat\theta_2\Big)$.

This pretty much looks like some invariance property argument and I think I can state it verbally: Step 1 yields the MLE of of the true value of $\theta_1$ because if $\hat\theta$ is the MLE then $\text{proj}_1(\hat\theta) = \hat\theta_1$ is the MLE of $\theta_1$ by the invariance principle. Since $\hat\theta_1(\theta_2)$ is a function of $\theta_2$, the same argument applies. That is, $\hat\theta_1(\hat\theta_2)$ is the MLE of $\hat\theta_1(\theta_2)$. But why does the MLE derived from the above steps coincide with $\hat\theta$? Is this result really a corollary from the invariance principle or is it a different theorem?

$\endgroup$
4
  • $\begingroup$ The answer depends on what precisely you mean by "argsup": could you please state your definition? $\endgroup$ – whuber Jun 25 '18 at 12:55
  • $\begingroup$ Since we are working most of the time with functions that posses a maximum, supremum became something like a synonym for maximum. So $$\hat\theta = \operatorname*{argsup}_\theta L(\theta) \iff L(\hat\theta)\geq L(\theta)$$ for each $\theta$ $\endgroup$ – Syd Amerikaner Jun 25 '18 at 13:39
  • $\begingroup$ I'm having a hard time understanding your question, then. In what sense could Step 1 "yield the MLE of the true value of $\theta_1$" when in fact it is producing a (likely non-constant) function instead of a number? $\endgroup$ – whuber Jun 25 '18 at 16:11
  • $\begingroup$ My question is actually wether partitioning the parameter vector and then do sequential maximazation can be regarded as an application or a corollary of the invariance principle. And if yes, how? (what you cited was just an attempt to prove why it may be true by playing around with some functions. Of course, this might be huge nonesense and thus giving you a hard time to understand. So please don't bother too much with my explanation. Maybe I should have emphasized the question more) $\endgroup$ – Syd Amerikaner Jun 25 '18 at 16:38
1
$\begingroup$

You question relates to the principle of the iterated supremum, which (loosely stated) says:

$$\sup_{\theta_1, \theta_2} L(\theta_1, \theta_2) = \sup_{\theta_2} \Big( \sup_{\theta_1} L(\theta_1, \theta_2) \Big).$$

This theorem says that you can find the supremum of a multivariate real function in an iterative manner, by taking the supremum one argument at a time. (Note that each time you take the supremum over an argument parameter, the result is a function of any remaining parameters.)

Now, if each of the supremum operations in your iteration has an argument value that gives the maximum$^\dagger$ (i.e., the $\text{argmax}$ exists at each step, yielding the resulting supremum), then you can obtain an analogous result for the $\text{argmax}$, and so you can then obtain the MLE via iteration. Of course, once you want to replace $\sup$ with $\text{argmax}$ things get a bit trickier, because the whole idea of the supremum is that it is well-defined (for real functions) even when there is not a maximising value. So if you want to formulate an analogous iterative property for the maximising argument, you need to restrict attention to functions that have well-defined maxima, which means that you are going to get a less general principle. Roughly speaking, you will find that this iterative method works to find the MLE in a wide class of cases (but not all of them).


$^\dagger$ I note that in your question you use the term $\text{argsup}$, which is really unclear (and whuber is right to question this in the comments). If there is an argument that attains the supremum then this is an $\text{argmax}$ in the usual sense; calling this $\text{argsup}$ just leads to confusion.

$\endgroup$
2
  • $\begingroup$ thanks for the answer and your comments (also @whuber) regarding the argsup statement. As I said, I got used to replace max by sup so I didn't really think about it as in most situation I encountered functions with well-defined maxima. To avoid further confusion a straight (follow-up) question: can the principle of iterated supremum be stated in terms of invariance principle of mle? $\endgroup$ – Syd Amerikaner Jun 29 '18 at 23:26
  • $\begingroup$ No, I don't think so. The invariance principle relates to the optimisation of composite functions, where you have an argument that is a function of another more primitive argument. $\endgroup$ – Ben Jun 30 '18 at 0:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.