Bernoulli random variable parameter estimation

Question

Suppose $\theta$ is the probability that a Bernoulli random variable is one (therefore $1-\theta$ is the probability that it's zero). I have a sequence of $n$ of these i.i.d. Bernoulli random variables, $m$ out of which are ones. It is a well-known fact that $\hat{\theta}=m/n$ is a maximum likelihood estimator (MLE) for $\theta$ (and it is also minimum-variance unbiased estimator).

Now, in my case, I know that $\theta=\frac{1}{2}+\frac{1}{2}e^{-x}$ for some unknown $x$, where $x$ satisfies $0<x<\infty$. This implies that $\theta$ satisfies $\frac{1}{2}<\theta<1$. I am trying to find an estimator for $x$ given $m$ and $n$ (and, perhaps the full vector of observed random variables). So, I try to find an MLE by first writing down the log-likelihood function:

$$\log\mathcal{L}(x;n,m)=-n\log 2+n\log(1-e^{-x})+m(\log(1+e^{-x})-\log(1-e^{-x}))$$

Then I take the derivative with respect to $x$ and set it to zero:

$$\frac{\partial\log\mathcal{L}(x;n,m)}{\partial x}=\frac{n}{e^x-1}-\frac{2m}{e^{-x}-e^{x}}=0$$

Solving for $x$ yields the estimator $\hat{x}=-\log(2m/n -1)$. However, if less than half the random variables are one (i.e. $m/n<1/2$ -- and this certainly can happen in the scenario that I described), then the estimator yields an imaginary number (due to a negative in the logarithm).

What did I do wrong? How does one estimate $x$ in this case?

EDIT

John A. Ramey pointed out that how for certain values of $\theta$, MLE leads to a degenerate answer. Perhaps there are methods other than MLE?

Answer 1

Instead of tedious derivations, simply invoke the invariance property of MLEs. Then solve for $x$ using basic algebra. However, note that this approach will lead to the same estimator you derived, i.e., $\hat{x}=-\log(2m/n -1)$.

So what to do? First, ignore estimation for the moment. Look at what the true value of $x$ would be if you knew the true value of $\theta$.

Suppose $\theta = 0$. Then, $e^{-x} = -1$, which implies that $x = - \log (-1)$. As you noted, this is not a real value.

So what about $\theta = 1/2$? We have that $e^{-x} = 0$, which is a problem for you yet again.

The point here is that with a couple of special cases based on known values, we can see that $x$ is undefined in a large number of cases. Your estimation is not wrong per se. The value of $x$ is simply undefined (not real) for certain values of $\theta$.

Answer 2

Since your knowledge about $\theta$ restricts the parameter space to $\Theta = (\tfrac{1}{2}, 1)$, you need to respect that when solving for the maximum likelihood. In other words, the maximum likelihood estimator is the solution to the constrained maximization problem, not the unconstrained maximization problem you've solved.

The log-likelihood you're interested in is

$$ \ln \mathcal{L}(\theta) = (n - m)\ln(1 - \theta) + m\ln\theta $$

and it turns out that it's concave almost everywhere, since

$$ \frac{d^2}{d\theta^2} \ln\mathcal{L}(\theta) = -\frac{n - m}{(1 - \theta)^2} - \frac{m}{\theta^2} < 0 \text{ for all } \theta \not\in \{0, 1\}. $$ As you've already pointed out, the unconstrained maximum is at $\theta_0 = m / n$. When this falls within your constraint $\Theta$, it is the maximum. When $\theta_0 \leq \tfrac{1}{2}$, the log-likelihood increases as $\theta$ moves towards $\theta_0$. However, the constraint requires that $\theta > \tfrac{1}{2}$, so the constrained maximum does not exist, and consequently, neither does the MLE. A similar situation occurs when $\theta_0 = 1$. Regardless of the actual value of $\theta_0$, the MLE does not exist because these situations are possible.

If you are willing to compromise and allow $x \in [0, \infty]$, so that $\Theta = [\tfrac{1}{2}, 1]$, then by the above it follows that the MLE of $\theta$ is

$$ \hat{\theta} = \max(\tfrac{1}{2}, \tfrac{m}{n}) $$

From a practical perspective, if you really need $x \in (0, \infty)$, you might instead assume $x \in [\epsilon, \ln\epsilon]$, where $\epsilon$ is a very small number. Alternatively, you could draw more observations; if your model is correct, eventually $\hat{\theta}$ should fall in $(\tfrac{1}{2}, 1)$. If it does not, you've either been hit by very bad luck, or you need to reconsider the validity of the model.

In situations where the log-likelihood is not so amenable to direct analysis, you could also solve for the MLE using constrained optimization techniques such as Lagrange multipliers.

Once you've found the MLE of $\theta$, then as John A. Ramey pointed out in his answer, you can invoke the invariance property of MLEs.

Edit: after thinking about this more, I've added a few more details.