4
$\begingroup$

Let $X_i$ be a binary random variable where $P(X_i = 1) = p$. I want to find the MLE estimator for $\theta = p(1-p)$.

The likelihood function should be $$ L(\theta) = \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i} = p^{\sum_{i=1}^nx_i}(1-p)^{n- \sum_{i=1}^nx_i}, $$ How should I proceed? Seems like I can not just take the derivative with respect to $\theta = p(1-p)$.

Edit: You use the plug in estimator, so you find $\hat{p}_{mle}$ first, then $\hat{\theta} = \hat{p}(1-\hat{p})$.

Now suppose I want to find the asymptotic distribution for $\hat{\theta}$. how do I compute the information matrix?

$\endgroup$
6
  • 1
    $\begingroup$ If this is part of a homework assignment, please edit your question to add the self-study tag. $\endgroup$
    – microhaus
    Commented May 26, 2023 at 18:23
  • $\begingroup$ 1. Consider taking log-likelihood rather than likelihood. 2. When computing partial derivatives of log-likelihood and setting to 0, do so with respect to $p$ and solve for the maximum likelihood estimator $\hat{p}$. 3. Now treat your variance $\theta = f(p)$ as a function of parameter. You can now plug-in $\hat{p}$ so that $\hat{\theta} = f(\hat{p})$. $\endgroup$
    – microhaus
    Commented May 26, 2023 at 18:39
  • $\begingroup$ Why Is this $\hat{p}(1- \hat{p})$ still a MLE? $\endgroup$ Commented May 26, 2023 at 18:45
  • $\begingroup$ Consider what happens if you choose a $\theta^* \neq \hat{p}(1-\hat{p})$. You are, implicitly, choosing an estimate of $p \neq \hat{p}$, i.e., one that is not the MLE. Therefore $\theta^*$ cannot be the MLE of $\theta$. $\endgroup$
    – jbowman
    Commented May 26, 2023 at 18:48
  • 2
    $\begingroup$ $\theta = p(1-p)$ does not fully specify the likelihood function. How would you define the MLE? $\endgroup$ Commented May 26, 2023 at 18:56

1 Answer 1

1
$\begingroup$

There is no MLE of $\theta$.

$$L(\theta) = \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i}$$

This expression of the likelihood function is incorrect. The left-hand side expresses the likelihood as a function of $\theta$, but the right hand side is a function of $p$.

You can not fix this either because $\theta$ is a non-injective function of $p$ and can not be inverted. Given $\theta$ we have two possible values of $p$

$$p=\frac{1}{2}\pm\sqrt{\frac{1}{4}-\theta}$$

so you can not compute the probability of the data given $\theta$ unless you have a second parameter.


how do I compute the information matrix?

If you would have a valid MLE then you could start with the information matrix of some parameter and apply a scaling according to the square of the derivative of the transformation.

However, note that some transformed variable can be biased and the Fisher Information alone is not an indication of asymptotic variance. See this example: Why the variance of Maximum Likelihood Estimator(MLE) will be less than Cramer-Rao Lower Bound(CRLB)?


The special case of finding the variance of the distribution of the statistic $\hat\theta = \hat{p}(1-\hat{p}) = \hat{p}-\hat{p}^2$ can be done more directly by computing $$\begin{array}{} Var(\hat\theta) &=& E[(\hat{p}-\hat{p}^2)] - E[\hat{p}-\hat{p}^2]^2 \\ &=& E[\hat{p}^4-2\hat{p}^3+\hat{p}^2] - E[\hat{p}-\hat{p}^2]^2\\ &=& E[\hat{p}^4]-2E[\hat{p}^3]+E[\hat{p}^2] - (E[\hat{p}]-E[\hat{p}^2])^2 \end{array}$$

which can be expressed in terms of the raw moments of $\hat{p}$ (a binomial distributed variable, scaled by $1/n$)

$$\begin{array}{} E[\hat{p}] &=& p \\ E[\hat{p}^2] &=& p^2 + \frac{p(1-p)}{n} \\ E[\hat{p}^3] &=& p^3 + 3 \frac{p^2(1-p)}{n} \\ E[\hat{p}^4] &=& p^4 + 6 \frac{p^3(1-p)}{n} + 3 \frac{p^2(1-p)^2}{n^2} \\ \end{array}$$

and the variance can be written as

$$Var(\hat\theta) = \frac{2p^4-4p^3+2p^2}{n^2} + \frac{-4p^4+8p^3-5p^2+p}{n}$$

where the second term becomes dominant for large $n$ and is the same result as using the Fisher Information matrix

$$p(1-p)/n \cdot \left(\frac{\text{d}\theta}{\text{d}p}\right)^2 = \frac{p(1-p)(1-2p)^2}{n}$$

In the case of $p=0.5$ this would lead to zero variance (or an infinite value in the information matrix). In that case you can still use the Delta method with a second order derivative as demonstrated in this question: Implicit hypothesis testing: mean greater than variance and Delta Method

$\endgroup$
11
  • $\begingroup$ I found this theorem from casella and Berger, page 320. If $\hat{\theta}$ is the MLE of $\theta$, then for any function $\tau(\theta)$, the MLE of $\tau(\theta)$ is $\tau(\hat{\theta})$. $\endgroup$ Commented May 26, 2023 at 23:04
  • $\begingroup$ @BudLightD.Va That is only true if the function is an injective function, which is not your case. $\endgroup$ Commented May 26, 2023 at 23:07
  • 1
    $\begingroup$ C&B called this the invariance property of MLE, if the function is injective, then this is pretty obvious. But this property still holds even if the function is not injective. $\endgroup$ Commented May 26, 2023 at 23:12
  • 1
    $\begingroup$ If you do it like that then the second part of my answer still holds. You get the same variance with a scaling factor $\left(\frac{\text{d}\tau}{\text{d}\theta}\right)^2$. You could see this intuitively with the delta method. (that's for the variance, I am not sure whether the Fisher information is still correctly defined when you use this alternative likelihood, the parameter is not complete) $\endgroup$ Commented May 26, 2023 at 23:23
  • 1
    $\begingroup$ An alternative interpretation of the question--and it's the one I favor--is that given a Binomial sample with unknown parameter $p,$ what is the MLE of $p(1-p)$ and what is its asymptotic distribution? That can be determined but it's not trivial. Thus, instead of responding to some faults in the statement of the question, I am suggesting responding to the underlying question itself--which is interesting. $\endgroup$
    – whuber
    Commented May 27, 2023 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.