1
$\begingroup$

Let $X_1,\cdots,X_n$ be an i.i.d sample from $P_\theta(X=x)=\theta^x(1-\theta)^{1-x}, x=0,1; 0 \le \theta \le \frac{1}{2}$. Its the method of moments estimator of the MLE better? Why?

My work:

I found the following two estimators:

$\hat{\theta}_{MoM}=\bar{X}$ with MSE: $\frac{\theta(1-\theta)}{n}$

$\hat{\theta}_{MLE}=min(\bar{X},1/2)$ with piece-wise MSE: $\frac{\theta(1-\theta)}{n}, \bar{X} < 1/2$ and $\theta^2-\theta+1/4, \bar{X} > 1/2$

How do I show from here which estimator is better?

Updated Work:

When $\bar{X} \le \frac{1}{2}, MSE(\hat{\theta}_{MLE})=MSE(\hat{\theta}_{MM})$

When $\bar{X} > \frac{1}{2}$, let $g(\theta)=MSE(\hat{\theta}_{MLE}) - MSE(\hat{\theta}_{MM}) = (1 + \frac{1}{n})\theta^2 - (1+\frac{1}{n})\theta + \frac{1}{4} \stackrel{\text{set}}{<} 0$

which is a positive quadratic equation, decreasing until $\theta = 1/2$.

By solving the quadratic equation, we get $\theta < \frac{1}{2}-\frac{1}{2\sqrt{n+1}} < \frac{1}{2}$.

So, $g(\theta) < 0$ for $\theta < \frac{1}{2}$. Therefore, we prefer MLE over method of moments estimator.

$\endgroup$
  • 2
    $\begingroup$ Theoretically MLE can be zero if all observations are zero. So if you take that into account your answer will be correct. MLE should be better because for one, it is guaranteed to take care of the restriction $[0,1/2]$ unlike the MOM estimator. Compare the two MSEs for a final answer. $\endgroup$ – StubbornAtom Feb 19 at 19:17
  • 2
    $\begingroup$ Shouldn't you compare them in terms of bias, efficiency or MSE? $\endgroup$ – Guilherme Marthe Feb 19 at 20:39
  • $\begingroup$ You are both right. @StubbornAtom. $\endgroup$ – DanielTheRocketMan Feb 20 at 2:18
  • 1
    $\begingroup$ In principle, there's no reason why you can't restrict a MoM estimator to lie within the known range of values for the parameter; if you did that, of course, the two estimators would be the same. $\endgroup$ – jbowman Feb 20 at 2:28
1
$\begingroup$

MLE of $\theta$ is more precisely given by

$$\hat\theta_{MLE}=\overline XI_{0\le \overline X\le \frac12}+\frac12I_{\overline X>\frac12}=\begin{cases}\overline X&,\text{ if }0\le\overline X\le \frac12 \\ \frac12&,\text{ if }\overline X>\frac12 \end{cases}$$

Method of moments estimator of $\theta$ is as you say $\hat\theta_{MOM}=\overline X$.

Therefore for $\theta\in[0,\frac12]$,

\begin{align} \operatorname{MSE}_{\theta}(\hat\theta_{MLE})&=\mathbb E_{\theta}(\hat\theta_{MLE}-\theta)^2 \\&=\sum_{0\le j\le \frac12}(j-\theta)^2\mathbb P_{\theta}(\overline X=j)+\sum_{j>\frac12}\left(\frac12-\theta\right)^2\mathbb P_{\theta}(\overline X=j) \end{align}

And

\begin{align} \operatorname{MSE}_{\theta}(\hat\theta_{MOM})&=\mathbb E_{\theta}(\overline X-\theta)^2 \\&=\sum_{0\le j\le \frac12}(j-\theta)^2\mathbb P_{\theta}(\overline X=j)+\sum_{j>\frac12}(j-\theta)^2\mathbb P_{\theta}(\overline X=j) \end{align}

So for every $\theta\in[0,\frac12]$,

\begin{align} \operatorname{MSE}_{\theta}(\hat\theta_{MLE})-\operatorname{MSE}_{\theta}(\hat\theta_{MOM})&=\sum_{j>\frac12}\left[\left(\frac12-\theta\right)^2-(j-\theta)^2\right]\mathbb P_{\theta}(\overline X=j) \end{align}

You just have to check the sign of this expression to conclude.

Similar question: How to show that $E[(\hat\theta -\theta)^2]<Var(\bar X)=\dfrac{1}{n}$?

$\endgroup$
  • $\begingroup$ I like this route, and the similar question was helpful. Thanks! I added some updated work in my post using the way that I was initially pursuing. Does my work make sense? I prefer your work, but for my sanity, I wanted to see if my original route would work, too. $\endgroup$ – Edison Feb 22 at 20:08
  • 1
    $\begingroup$ @Edison MLE is at least as good as MOM in terms of MSE if $\operatorname{MSE}_{\theta}(\hat\theta_{MLE})-\operatorname{MSE}_{\theta}(\hat\theta_{MOM})\le 0$ for all $\theta\in[0,1/2]$ (MLE is better if the inequality is strict for every $\theta$). You can show this by calculating the respective MSE's of course but that is quite unnecessary as you can see in this answer. There is virtually no calculation required apart from checking the sign of $\left(\frac12-\theta\right)^2-(j-\theta)^2$ for $j>\frac12$ and $\theta\in[0,1/2]$. $\endgroup$ – StubbornAtom Feb 23 at 16:18
  • $\begingroup$ I see. So $(1/2 -\theta)^2 - (j-\theta)^2=1/4-\theta-j^2+2j\theta=(1/4-j^2)-\theta(1-2j) < 0$, since both terms are $<0$. This is an elegant solution. Is this a common tactic when working with piecewise MLEs? Also, how do you know which case for the MLE gets the equality? $\endgroup$ – Edison Feb 24 at 1:53
  • $\begingroup$ @Edison When parameter space is restricted, the restricted MLE will be equal to the unrestricted MLE whenever the latter falls within the restriction. If not, restricted MLE will be a boundary point. $\endgroup$ – StubbornAtom Feb 24 at 15:34
1
$\begingroup$

If your sample is large and your data is independent and identically distributed, you may assume that all the assymptotic properties of the maximum likelihood estimator are valid: (1) Consistency; (2) Functional invariance; (3) Efficiency; (4) Assymptotic mean square error. See Wikipedia for details or Introduction to the Mathematical andStatistical Foundations of Econometrics - Bierens.

Unfortunatelly, the maximum likelihood estimator has no finite sample properties. I am not sure if it is possble to derive finite sample properties for the method of moments in the case, but usually if the sample is small, you can compare both estimators using Monte Carlo simulations.

I did a Monte Carlo simulation to compare the results. I basically found that if $\theta$ is small, the error of both estimators is the same. On the other hand, if $\theta$ is close to 0.5, then the ML works better since $\theta$ is limited by 0.5 and the value of the parameter is never larger than 0.5.

import matplotlib.pyplot as plt
import numpy as np



def evalMSE(estimatedParameter,realParameter):
    mse=0
    for i in range(len(estimatedParameter)):
        mse=mse+(estimatedParameter[i]-realParameter)**2
    return mse    

def generateData(theta,sampleSize):
    sample=np.empty([sampleSize])
    for i in range(sampleSize):
        x=np.random.uniform(0,1)
        if(x<1-theta):
            sample[i]=0
        else:
            sample[i]=1
    return sample


def MM(data):
    return np.mean(data)

def ML(data):
    return np.min([np.mean(data),0.5])

if __name__=="__main__":
    numberSamples=1000
    sampleSize=100

    vectorML=np.empty([numberSamples])
    vectorMM=np.empty([numberSamples])

    theta=0.01
    for i in range(0,numberSamples):
        data=generateData(theta,sampleSize)
        vectorMM[i]=MM(data)
        vectorML[i]=ML(data)
    mseMM=evalMSE(vectorMM,theta)    
    mseML=evalMSE(vectorML,theta)
    plt.figure(num=1)        
    plt.hist(vectorMM,bins=30,density=True)
    plt.figure(num=2)
    plt.hist(vectorML,bins=30,density=True)
    print(mseMM)
    print(mseML)

For $\theta=0.4$ We get

mseMM
Out[5]: 2.466999999999995

mseML
Out[6]: 2.3957999999999933

The distributions of $\theta$ are shown below:

$\theta$ distribution by MM:

Distribution of theta by MM

$\theta$ distribution by ML:

Distribution of theta by ML

$\endgroup$
  • 2
    $\begingroup$ The question requests comparing the two estimators in terms of MSE: that is, the basis for comparison is quadratic loss. $\endgroup$ – whuber Feb 20 at 0:08
  • 1
    $\begingroup$ Yes, I edited the question. $\endgroup$ – DanielTheRocketMan Feb 20 at 2:11
  • 1
    $\begingroup$ Thank you for providing a more computational approach to solving this problem. I will play around with some of this code this evening. Cheers! $\endgroup$ – Edison Feb 22 at 20:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.