1
$\begingroup$

This is from Robert Hogg's Introduction to Mathematical Statistics 6th Edition Exercise 6.1.13. The question is:

Let $X_{1},X_{2},...,X_{n} $ be a random sample from a distribution on $\mathbb{R}$ with one of two pdfs: $$f(x;\theta)=\begin{cases}\dfrac{\exp\{-x^2/2\}}{\sqrt{2\pi }}&\text{ if }\theta=1\\\dfrac{1}{\pi(1+x^2)}&\text{ if }\theta=2\end{cases} $$ Find the mle of $\theta$.

My simple solution is $\hat{\theta}=ArgmaxL(\theta;X)=ArgmaxL(1,2;X)$.

The max value for $\theta=1$ is $\frac{1}{\sqrt{2\pi}}$, when x=0 for the standard normal distribution

The max value for $\theta=2$ is $\frac{1}{\pi}$, when x=0 for the cauchy distribution.

Therefore, mle of $\theta$ is $\frac{1}{\pi}$

I am not sure whether my solution is correct or not. Please check my solution. Thank you very much

Sorry, the mle should be $\frac{1}{\sqrt{2\pi}}$ since it is bigger than $\frac{1}{\pi}$ by my simple solution.

After poder this problem for two days, I think the mle is "0" since by difination of MLE

$\hat{\theta}=ArgmaxL(\theta;X)$ which means that $L(\theta;X)$ achieve its maximum value at $\hat\theta$. Since the likelyhood fucntion can achive its maximum value at 0, therefore, by the defination, the mle is 0.

Further, the mle can be a statistic $\overline{X}$ which is just a function of data, then why we can not also treat constant 0 as a fucntion of data, which is 0*data, i.e 0*X.

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ When taking the sample how is it decided which $\theta$ is used? $\endgroup$
    – mpiktas
    Apr 6, 2015 at 14:19
  • 1
    $\begingroup$ @mpiktas: $\theta$ is a model index, it does not have to be random. $\endgroup$
    – Xi'an
    Apr 6, 2015 at 14:45
  • 3
    $\begingroup$ Your solution is not correct. You are given a sample, you don't choose what values it takes. $\endgroup$
    – Glen_b
    Apr 6, 2015 at 14:47
  • 2
    $\begingroup$ The MLE is either $1$ or $2$, depending on which likelihood is larged for the observed sample, so I do not understand why you pick a single $x$ and let it be equal to zero. $\endgroup$
    – Xi'an
    Apr 6, 2015 at 14:47
  • 3
    $\begingroup$ The MLE is provided by$$\hat\theta=\arg\max_{\theta\in\{1,2\}} L(\theta|x_1,\ldots,x_n)$$ $\endgroup$
    – Xi'an
    Apr 6, 2015 at 14:49

2 Answers 2

Reset to default
4
$\begingroup$

It appears that the only issue with the answer the OP gave in the question is that he has overlooked the fact that we have a sample of size $n$ in our hands. Then what the MLE maximizes is the Likelihood of the sample. Since this is a random (=i.i.d) sample, it follows that the joint density of the sample is the product of $n$ densities, and in turn it appears that the likelihood of the sample is the joint density expression viewed as a function of the unknown parameter, for the given sample:

$$L(\theta|x_1,\ldots,x_n)=\begin{cases} \left(\frac{1}{\sqrt{2\pi }}\right)^n\cdot\exp\big\{-\sum_{i=1}^n(x_i^2/2) \big\} &\text{ if }\theta=1\\ \\ \left(\frac{1}{\pi}\right)^n\prod_{i=1}^n(1+x_i^2)^{-1}&\text{ if }\theta=2\end{cases}$$

where the $x_i$'s are the actual series of numbers available (realizations of the RV's), and they are to be treated as fixed numbers, much like $\pi$ for example.

But wait, does the above look like a function of $\theta$ given the sample? It seems more like "conditional on the value of the argument, the function is..."

Let's see: for us mortals, a function is defined by two things: its domain, and its functional form. If we want $\theta$ to be its argument, then its domain is $\{1,2\}$ and its functional form changes as its domain changes. That's perfectly fine, we have a case of a "piece-wise" function. These functions may have maxima and minina, etc. as any other function.

Since the domain (the parameter space) is constrained a priori to only two values, what you have to do is evaluate the two branches for the given $x_i$'s and the one with the larger value will be the function's maximum. Since each branch is uniquely associated with a single value of $\theta$, this $\theta$ will be the $\text {argmax}$ of the function. And since this function is a likelihood, then you can argue that you just performed maximum likelihood estimation related to the unknown parameter $\theta$, even though $\theta$ itself does not appear inside the functional forms. Note that in this approach, the constants are indispensable, since they too affect the value of the likelihood. Calculations become simpler if we consider the log-likelihood (without omitting the constants) which is a monotonic transformation,

$$\ln L(\theta|x_1,\ldots,x_n)=\begin{cases} -n\ln \left(\sqrt{2\pi }\right)-(1/2)\sum_{i=1}^nx_i^2 &\text{ if }\theta=1\\ \\ -n\ln \pi - \sum_{i=1}^n \ln(1+x_i^2)&\text{ if }\theta=2\end{cases}$$

For the above to be valid inference, it has to be the case that if, say, the $x_i$'s available have in reality been drawn from a Standard Normal distribution, then if we plug the specific series of $x_i$'s into the Cauchy sample likelihood, we will obtain a smaller numerical value than if we plug them into the Standard Normal sample likelihood. Will it? And, moreover will we obtain the correct result always or as a probabilistic event possibly seeing its probability increasing as the sample size increases?

Let's simulate to obtain some evidence. I created i.i.d. samples of sizes $n =50,100,500,1000$ drawn from a standard normal distribution. For each sample size, I generated $10,000$ such samples. For each sample I calculated the two values of the log-likelihood, and then I obtained the empirical percentage of times the value of the Standard Normal log-likelihood was greater than the value of the Cauchy log-likelihood, i.e. the percentages of times the procedure described above gave me the correct answer. This percentage approximates the probability of obtaining a correct answer. Denote this event as

$$B = \{\text{sample is normal and the value of the standard Normal log-likelihood} $$

$$\text{was greater than the value of the standard Cauchy log-likelihood}\}$$

I obtained

\begin{array}{| r | r | r | r|} \hline \text{n} & \;\;\text{% of A } \\ \hline \hline 50 & 100.00 \\ \hline 100 & 100.00 \\ \hline 500 & 100.00 \\ \hline 1000 & 100.00 \\ \hline \end{array}

One obtains the analogous result if one tries the corresponding procedure using Cauchy samples (to be honest, here I got $2$ false results out of $10,000$ when the sample sizes was $n=50$).

Any ideas as to why we get results with such certainty?

Improve this answer
$\endgroup$
6
  • 3
    $\begingroup$ Cauchy and Normal distributions are so far apart that it takes only a few iid samples to tell the difference. Indeed, almost any Cauchy sample of size $n\ge 50$ is so likely to have a really extreme value that its presence makes the Normal astronomically unlikely (because the Normal tails decay so rapidly). BTW, I would have to say that very definitely yes your formulas look like functions of $\theta$: all its possible values are explicitly written on the right hand sides! $\endgroup$
    – whuber ♦
    Apr 6, 2015 at 22:14
  • $\begingroup$ Now I think the mle is 0 $\endgroup$
    – Deep North
    Apr 9, 2015 at 12:54
  • $\begingroup$ @DeepNorth Why don't you actually perform the maximization of the likelihood as you understand it, to see whether you do obtain that the MLE is zero? It would help if you included in your question the detailed maximzation procedure that you will implement. $\endgroup$
    – Alecos Papadopoulos
    Apr 9, 2015 at 13:05
  • $\begingroup$ I think if plug 0 into the likely hood fuction, it will achieve the maximum value either it is standard normal or stadard cauchy. The problem is whether we can treat 0 as an function of data. $\endgroup$
    – Deep North
    Apr 9, 2015 at 13:37
  • 2
    $\begingroup$ Stating that "the MLE is $0$" is like responding to the question "which would you prefer, apples or cherries?" with the answer "oranges." Since there are only two possible values of $\theta$ in this problem, the MLE had better be one of them! $\endgroup$
    – whuber ♦
    Apr 9, 2015 at 14:15
1
$\begingroup$

Hi， Alecos Papadopoulos， thank you very much. I did some simulations, the results showed that the log likelihood function of standard normal is always bigger than log likelihood function of Cauchy. then I think I should choose $\theta=1$ as the mle of the "piece wise" function. The following is the r fuction I wrote to perform the simutions.

sim_mle<-function(nsim){
a<-c()
b<-c()
d<-c()
e<-c()
f<-c()
g<-c()

count<-0
count2<-0

for(i in 1:nsim){
x<-rnorm(10*i,0,1)
n<-length(x)
a[i]<-n*log(sqrt(2*pi))-(1/2)*sum(x^2)
b[i]<--n*log(pi)-sum(log(1+x^2))
d[i]<-a[i]-b[i]
if(d[i]>=0){count=count+1}
p1<-count/nsim

y<-rcauchy(100*i,0,1)
m<-length(y)
e[i]<-m*log(sqrt(2*pi))-(1/2)*sum(y^2)
f[i]<--m*log(pi)-sum(log(1+y^2))
g[i]<-e[i]-f[i]
if(g[i]>=0){count2=count2+1}
p2<-count2/nsim;

}

return(list(norm_big=p1,norm_less=p2))
}
sim_mle(100)
sim_mle(1000)
Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ As I final remark, I have the impression that your initial confusion came from the fact that you were attempting to maximize the likelihood with respect to the $x$'s. But to obtain the MLE one should maximize the likelihood with respect to the parameter (i.e. treating the parameter as the argument of the function), while the $x$'s are treated as known parameters (fixed numbers). $\endgroup$
    – Alecos Papadopoulos
    Apr 11, 2015 at 18:32
  • $\begingroup$ Yes, lack of $\theta$ in the function form make me confused. $\endgroup$
    – Deep North
    Apr 11, 2015 at 23:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.