Consider a case that we see {H,H} in two coin tossing. We model this with binomial distribution as: $P(D|x) = x^2*(1-x)^0 = x^2$ Now, if we want to compute x based on MLE: $Log(P(D|x)) = 2Logx$. and the derivative is: $\frac{2}{x}$. D is Data and x is the parameter. So, what is the x?

  • 1
    $\begingroup$ All you need to do is maximize $x^2$ for $0\le x\le 1$. That doesn't require differential calculus for its solution! $\endgroup$ – whuber Oct 27 '17 at 19:51
  • $\begingroup$ That is true @whuber, but why this exception occurs (based on the formula)? $\endgroup$ – keramat Oct 27 '17 at 19:58
  • 2
    $\begingroup$ To what formula do you refer? Note that not every function is differentiable everywhere, so you need always to remember that inspecting zeros of a derivative is only the first step of a more systematic investigation. It cannot be replaced by a single formula (at least not without making restrictive assumptions that rule out your situation). $\endgroup$ – whuber Oct 27 '17 at 20:21

If I understand your question correctly, you are concerned with the problem of maximizing the likelihood in the case where (without being Bayesian and having a prior) the observation will lead to an estimate on the boundary of the parameter space.

In the frequentist setting I think this is a way to deal with this:

Don't allow the parameter to take values on the boundary and simply conclude that you don't have enough data if the MLE lands there (probably a wise course in this case). This is related to the idea of "asymptotically well-defined" estimators, i.e., estimators where the probability of obtaining an estimate outside of the parameter space goes to zero. In other words, unless the coin is heads on both sides (P(Tails = 0)), you will eventually observe a tails and your estimate will be in the parameter space, i.e., the open interval (0,1).

However, with parameter space [0,1], you can still maximize the likelihood, as was pointed out by @whuber. The MLE will then be $\hat{p} = 1$, if $p = P(X=\text{Tails})$.

The question about "the formula" presumably relates to the fact that we like to solve the equation where we set the score function equal to zero ($\frac{\partial \log L(\theta ; y)}{\partial \theta} = 0$) and solve for $\theta$ to obtain a stationary point which is then $\hat \theta$. But that implicitly leans on convex optimization: (1) the log-likelihood being convex (which you demonstrate by checking the second derivative being strictly positive) and (2) the stationary point not being on the boundary of the parameter space. In this case, (2) is broken. In fact, if you consider the likelihood only as a function on the parameter space, the derivative isn't even defined there, only the derivative from the right in 0 and from the left in 1.

Lastly note that while a Bayesian prior can get your posterior distribution away from the boundary, your example would still be concerning because the choice of prior will then be pretty important in how the posterior looks like. Of course, to my mind, if you don't believe in your prior you have no business using it.

  • 1
    $\begingroup$ +1 it's a pleasure to read this thoughtful answer. Welcome to our site! $\endgroup$ – whuber Oct 27 '17 at 20:58
  • $\begingroup$ Thanks @Ketil B T, Why you are using L(θ;y) not L(y|θ)? $\endgroup$ – keramat Oct 28 '17 at 3:21
  • 1
    $\begingroup$ @keramat : Well, if you want to use the "conditional" notation, you should probably write $L(\theta \vert y)$. You can also simply write $L(\theta)$. I agree that my notation might be non-standard but I wanted to include the dependence of the likelihood on the realised observation. In my (Danish) training we used $L_y (\theta)$ but I have never seen that anywhere else, so did not want to confuse matters. Good intentions... :) $\endgroup$ – Ketil B T Oct 28 '17 at 15:42
  • $\begingroup$ @Ketil B T, based on some books, it should be p(D|θ). L(θ|y) is a MAP estimation. $\endgroup$ – keramat Oct 28 '17 at 16:55
  • $\begingroup$ Keramat, Many, many different communities, books, journals, and authors use shared statistical concepts but routinely adopt different notations for them; and they sometimes use the same notation in different ways. It is pointless to insist on one particular notation and erroneous to suppose it universally has the meaning you are familiar with. If you don't understand what Ketil has written, then just ask for clarification. $\endgroup$ – whuber Oct 28 '17 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.