I was wondering how ML is defined when the parameter does not completely specify the model. More concretely, suppose $X_1, X_2, \cdots, X_n$ are drawn iid such that $P(X_1=i)=\theta_i$, $ 1 \leq i \leq k$. I want to find the ML estimate of $\phi= \max_{1 \leq i \leq k} \theta_i$. To me it is not even clear if the ML estimate of $\phi$ is well defined in this case. Am I missing something here?

  • 2
    $\begingroup$ Take a look at the Categorical distribution. $\endgroup$ – Daniel Sep 12 '14 at 21:53
  • $\begingroup$ The categorical distribution I mentioned here is just an example. My question basically is, is ML estimation of a parameter not defined if my parameter does not fully characterise the log likelihood. $\endgroup$ – Devil Sep 13 '14 at 0:23

It's not about MLE vs Method of Moments or some other method, it's about proper reparameterization. If you want to reparameterize, you should be able to compute the values of old parameters, $\theta_i$, in terms of new parameters. You introduced one new parameter, $\phi$, which is the mode, and you need $(k-1)$ more. As far as I know, it's impossible to reparameterize the multinomial distribution using the mode.

  • $\begingroup$ So ML is not defined if I can not re-parametrise the log likelihood in terms of the parameter of interest? $\endgroup$ – Devil Sep 12 '14 at 23:45
  • $\begingroup$ Sure, you can't maximize what you don't know how to define. In such cases of non-existent closed-form solution, bootstrap can help. $\endgroup$ – James Sep 13 '14 at 3:16

I agree the question is not well defined at present; you haven't defined what the model is with respect to the new parameter $\phi$, so it doesn't make sense to ask about ML estimates.

We can ask about ML estimates of parameters that don't completely specify the model when we know what the model is (think ML estimate of Gaussian mean) and even parameters we don't directly observe (latent variables) using techniques such as the EM algorithm. We can also talk of constrained parameters, although I would argue in this case you would be better using a Bayesian approach. What we can't do is ask about parameters of some unknown model.

From the looks of it, you're half way to defining some sort of hierarchal model; you need to specify what role $\phi$ plays in your model, then you can ask about its ML estimate.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.