Suppose $\theta$ is the probability that a Bernoulli random variable is one (therefore $1-\theta$ is the probability that it's zero). I have a sequence of $n$ of these i.i.d. Bernoulli random variables, $m$ out of which are ones. It is a well-known fact that $\hat{\theta}=m/n$ is a maximum likelihood estimator (MLE) for $\theta$ (and it is also minimum-variance unbiased estimator).

Now, in my case, I know that $\theta=\frac{1}{2}+\frac{1}{2}e^{-x}$ for some unknown $x$, where $x$ satisfies $0<x<\infty$. This implies that $\theta$ satisfies $\frac{1}{2}<\theta<1$. I am trying to find an estimator for $x$ given $m$ and $n$ (and, perhaps the full vector of observed random variables). So, I try to find an MLE by first writing down the log-likelihood function:

$$\log\mathcal{L}(x;n,m)=-n\log 2+n\log(1-e^{-x})+m(\log(1+e^{-x})-\log(1-e^{-x}))$$

Then I take the derivative with respect to $x$ and set it to zero:

$$\frac{\partial\log\mathcal{L}(x;n,m)}{\partial x}=\frac{n}{e^x-1}-\frac{2m}{e^{-x}-e^{x}}=0$$

Solving for $x$ yields the estimator $\hat{x}=-\log(2m/n -1)$. However, if less than half the random variables are one (i.e. $m/n<1/2$ -- and this certainly can happen in the scenario that I described), then the estimator yields an imaginary number (due to a negative in the logarithm).

What did I do wrong? How does one estimate $x$ in this case?


John A. Ramey pointed out that how for certain values of $\theta$, MLE leads to a degenerate answer. Perhaps there are methods other than MLE?

  • 2
    $\begingroup$ Please notice that setting the derivative to zero only finds the critical points: to optimize the function, it is essential that you also evaluate it along the boundary of the domain of definition. In the case $m/n\lt 1/2,$ this means the MLE estimate of $x$ is $\hat{x}=\infty$. $\endgroup$ – whuber Sep 8 '13 at 1:19
  • $\begingroup$ There are many methods other than MLE. MLE is often used because of its nice large sample ($n \to \infty$) properties, but there are also methods focused on small sample properties, such as uniformly minimum variance unbiased estimators (UMVUE), Bayes estimators, and minimax estimators. Some of these may coincide with the MLE for your particular problem. Lehmann & Casella's Theory of Point Estimation gives a thorough account of these methods of estimation. $\endgroup$ – Nick Sep 13 '13 at 4:04

Instead of tedious derivations, simply invoke the invariance property of MLEs. Then solve for $x$ using basic algebra. However, note that this approach will lead to the same estimator you derived, i.e., $\hat{x}=-\log(2m/n -1)$.

So what to do? First, ignore estimation for the moment. Look at what the true value of $x$ would be if you knew the true value of $\theta$.

Suppose $\theta = 0$. Then, $e^{-x} = -1$, which implies that $x = - \log (-1)$. As you noted, this is not a real value.

So what about $\theta = 1/2$? We have that $e^{-x} = 0$, which is a problem for you yet again.

The point here is that with a couple of special cases based on known values, we can see that $x$ is undefined in a large number of cases. Your estimation is not wrong per se. The value of $x$ is simply undefined (not real) for certain values of $\theta$.

  • 1
    $\begingroup$ Very true. This is actually one of the big arguments that Bayesians use against frequentist is that the MLE and properties of minimum variance and unbiasedness can lead to very poor estimators. $\endgroup$ – user25658 Sep 7 '13 at 5:06
  • $\begingroup$ @John Hmmm... interesting point. I see what you are saying. However, suppose that I also know that $1/2<\theta<1$, i.e. $x$ is a positive and finite real number. Can I estimate it given the observed frequency of ones $m/n$? If not, can I somehow estimate $x$ using the full vector of observations of the random variables? (I'll adjust the question). $\endgroup$ – M.B.M. Sep 7 '13 at 5:18
  • 1
    $\begingroup$ Sure. But you need to restrict the support of $\theta$ in the likelihood, or as @BabakP mentioned, a prior could help here to regularize the estimation so that it is well-posed. $\endgroup$ – ramhiser Sep 7 '13 at 5:26

Since your knowledge about $\theta$ restricts the parameter space to $\Theta = (\tfrac{1}{2}, 1)$, you need to respect that when solving for the maximum likelihood. In other words, the maximum likelihood estimator is the solution to the constrained maximization problem, not the unconstrained maximization problem you've solved.

The log-likelihood you're interested in is

$$ \ln \mathcal{L}(\theta) = (n - m)\ln(1 - \theta) + m\ln\theta $$

and it turns out that it's concave almost everywhere, since

$$ \frac{d^2}{d\theta^2} \ln\mathcal{L}(\theta) = -\frac{n - m}{(1 - \theta)^2} - \frac{m}{\theta^2} < 0 \text{ for all } \theta \not\in \{0, 1\}. $$ As you've already pointed out, the unconstrained maximum is at $\theta_0 = m / n$. When this falls within your constraint $\Theta$, it is the maximum. When $\theta_0 \leq \tfrac{1}{2}$, the log-likelihood increases as $\theta$ moves towards $\theta_0$. However, the constraint requires that $\theta > \tfrac{1}{2}$, so the constrained maximum does not exist, and consequently, neither does the MLE. A similar situation occurs when $\theta_0 = 1$. Regardless of the actual value of $\theta_0$, the MLE does not exist because these situations are possible.

If you are willing to compromise and allow $x \in [0, \infty]$, so that $\Theta = [\tfrac{1}{2}, 1]$, then by the above it follows that the MLE of $\theta$ is

$$ \hat{\theta} = \max(\tfrac{1}{2}, \tfrac{m}{n}) $$

From a practical perspective, if you really need $x \in (0, \infty)$, you might instead assume $x \in [\epsilon, \ln\epsilon]$, where $\epsilon$ is a very small number. Alternatively, you could draw more observations; if your model is correct, eventually $\hat{\theta}$ should fall in $(\tfrac{1}{2}, 1)$. If it does not, you've either been hit by very bad luck, or you need to reconsider the validity of the model.

In situations where the log-likelihood is not so amenable to direct analysis, you could also solve for the MLE using constrained optimization techniques such as Lagrange multipliers.

Once you've found the MLE of $\theta$, then as John A. Ramey pointed out in his answer, you can invoke the invariance property of MLEs.

Edit: after thinking about this more, I've added a few more details.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.