1
$\begingroup$

In class, we talked about finding Maximum Likelihood Estimators but there is something I don't think we talked about. How is the MLE different for distributions with constrained bounds?

For example, for an iid random sample $X_1,\ldots,X_n$, if we compare the MLE for $\theta$ from a Bernoulli distribution versus the MLE for $\theta$ from another Bernoulli distribution but with 0 < $\theta$ < $\frac 12$, how is the MLE affected?

I know the MLE for a Bernoulli($\theta$) with 0 < $\theta$ < 1 is $\frac{\Sigma x_i}{n}$.

I thought it was the same for the 0 < $\theta$ < $\frac 12$ case but thinking back to a Uniform distribution, the MLE will vary depending on the bounds. However, I don't think order statistics is helpful here. Does anyone have any ideas?

Improve this question
$\endgroup$
3
  • $\begingroup$ Difference: If the parameter space is restricted compared with the natural parameter space, the maximum may occur on the boundary of the restricted space in which case it does not solve the likelihood equations, ie does not cancel the score function. $\endgroup$
    – Xi'an
    Mar 12, 2018 at 20:28
  • 1
    $\begingroup$ Part of the answer is trivial: you maximize the likelihood subject to the constraints. That's what MLE means. The non-trivial part often requires research: how do you assess the precision of estimates when the MLE lies on the boundary of the parameter space? Ordinarily this is done with likelihood ratios and a chi-squared approximation, but these approximations usually fail for constrained solutions. I would hope that answers not be limited to just the trivial aspect of the question. $\endgroup$
    – whuber ♦
    Mar 12, 2018 at 20:34
  • $\begingroup$ stats.stackexchange.com/q/151655/119261 $\endgroup$
    – StubbornAtom
    Jul 4, 2020 at 18:08

1 Answer 1

Reset to default
2
$\begingroup$

In general, but not always, what will happen is that the constrained MLE will be the closest possible value to the unconstrained MLE. To re-use your example, if $\sum x_i / n = 0.7$ but $0 < \theta \leq 0.5$, then the unconstrained MLE for $\theta$ is $0.7$ but the constrained MLE for $\theta$ is 0.5. This will always be the case if the log likelihood is concave in the parameter of interest.

Note that if you have a strict inequality constraint, for example, $0 < \theta < 0.5$, the constrained MLE doesn't exist, as there is no largest number less than $0.5$.

In some cases, though, the likelihood function can be multimodal. (The Student-t distribution is an example of this.) The unconstrained MLE corresponds to whichever mode is the highest (or, in the case of ties, is any value from the set corresponding to the highest modes.) In these cases, it may be that the constraint excludes the highest mode, so that the constrained MLE becomes EITHER the value corresponding to the highest mode remaining in the interior of the set of feasible values OR a value on the boundary of the set of feasible values - which latter may not appear to be a mode when looking at the unconstrained problem.

The following graph shows an example of this. The unconstrained MLE is at $6.4$, but, if we add a constraint at the red line ($5.5$), the new MLE is at $5.5$, despite the presence of a mode at $0.7$.

enter image description here

Improve this answer
$\endgroup$
2
  • $\begingroup$ In case the constrained MLE does not exist, shall we conclude that the unconstrained MLE is the actual MLE? $\endgroup$
    – StubbornAtom
    May 2, 2018 at 21:10
  • 1
    $\begingroup$ How can the constrained MLE not exist but the unconstrained MLE exist? (Serious question, not rhetorical.) Also, if the constraints are real, e.g., the mean # of traffic accidents on the Bay Bridge per day $>0$, the MLE should be subject to the constraint; it's just making sure the parameter estimate lies within the region of possibility, so to speak. Well, that's a bad example, but I hope you see what I mean - the real MLE is the constrained one, not the unconstrained one. $\endgroup$
    – jbowman
    May 2, 2018 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.