2
$\begingroup$
  • A property of the Maximum Likelihood Estimator is, that it asymptotically follows a normal distribution if the solution is unique.
  • In case of a continuous Uniform distribution the Maximum Likelihood Estimator for the upperbound is given through the maximum of the sample $X_i$.

I have a hard time figuring out how the distribution of the maximum converges in distribution to a Gaussian.

In the following Question it is claimed that the maximum of the sample $X_i$ of a $U[0,\theta]$ , with $\theta$ = 1, will follow a Beta distribution. Question about asymptotic distribution of the maximum

I also tried to figure it out empirically and always came to a more or less the result in the Graph bellow. Also from a logical point of view (atleast my logic) the distribution should never be able to converge to a Gaussian since the Expected Value of $\hat\theta$ is asymptotically equal to $\theta$ and because all possible $X_i$ have to be smaller than $\theta$, therefore there can not exist Values on the right side of $E[\hat\theta]$, which makes it impossible to converge to a Normal Distribution.

Where do I make my mistake? I haven't found a similiar question considering the contradiction.

enter image description here enter image description here

$\endgroup$
  • 1
    $\begingroup$ Your first bullet item is not universally true. Check out the assumptions needed to prove it and see which one(s) do not hold in your case. (I cannot tell you the answer because your term "uniform distribution" is ambiguous. It appears you might mean the family of uniform distributions on the intervals $[0,\theta],\theta\gt 0.$) $\endgroup$ – whuber Nov 12 '19 at 21:51
  • 3
    $\begingroup$ You have included conclusions but you haven't stipulated all the assumptions on which they rely. To see what can go wrong, suppose there are only two possible underlying distributions identified by parameter values $\theta\in\{0,1\}.$ Since the estimator will therefore be a value in $\{0,1\},$ there's no way it possibly could be asymptotically Normal, contradicting the highlighted bullet in the quoted text. As another example, consider estimating the mean of a Normal$(\mu,1)$ distribution where $0\le\mu.$ When $\mu=0,$ there's a 50% chance the MLE will equal $0:$ what happens asymptotically? $\endgroup$ – whuber Nov 12 '19 at 22:07
  • 2
    $\begingroup$ Re "only the case:" that's not so. To conclude asymptotic Normality, one uses a version of the Central Limit Theorem. This requires so-called "regularity" conditions required to make the log likelihood look like a sum of iid random variables. These conditions typically require (1) all distributions have common support (ruling out your example); (2) the true parameter is in the interior of an open interval of possible parameters (ruling out my examples); (3) the Fisher Information is positive-definite; and (4) the likelihood is sufficiently highly differentiable to apply Calculus. $\endgroup$ – whuber Nov 12 '19 at 22:54
  • 3
    $\begingroup$ @Mauro Can you please edit your question to incorporate the corrections required (e.g. things like what you say in comments that you "should have said") $\endgroup$ – Glen_b -Reinstate Monica Nov 12 '19 at 22:55
  • 1
    $\begingroup$ @whuber Thanks a lot! So the statement in the script is not sufficient, because the regularity conditions also have to be fullfilled. $\endgroup$ – Mauro Schläpfer Nov 13 '19 at 7:17
6
$\begingroup$

A property of the Maximum Likelihood Estimator is, that it asymptotically follows a normal distribution if the solution is unique.

Not necessarily. So far as I am aware, all the theorems establishing the asymptotic normality of the MLE require the satisfaction of some "regularity conditions" in addition to uniqueness. Roughly speaking, these regularity conditions require that the MLE was obtained as a stationary point of the likelihood function (not at a boundary point), and that the derivatives of the likelihood function at this point exist up to a sufficiently large order that you can take a reasonable Taylor approximation to it. (The proofs of asymptotic normality then use the Taylor expansion and show that the higher order terms vanish asymptotically.)

The notes you have shown in your question gloss over this requirement, so I imagine that your teacher is interested in giving you the properties for the general case, without dealing with tricky cases where the "regularity conditions" do not hold. However, if you have a look at textbooks that actually prove the asymptotic normality of the MLE, you will see that the proof always hinges on these regularity conditions. (And indeed, good textbooks will usually supply counter-examples that show that asymptotic normality does not hold for some examples that don't obey the regularity conditions; e.g., the MLE of the uniform distribution.)

In the case of the MLE of the uniform distribution, the MLE occurs at a "boundary point" of the likelihood function, so the "regularity conditions" required for theorems asserting asymptotic normality do not hold. So far as I am aware, the MLE does not converge in distribution to the normal in this case.

$\endgroup$
  • $\begingroup$ +1. But in this example $\hat\theta$ is not at a boundary point: the likelihood function is defined on $(0,\infty)$ and almost surely $\hat\theta$ is nonzero. The assumption that fails here is that the distributions in the family must have common support. Since they don't, we obtain "extra" information from the data: namely, each value $x_i$ definitively rules out the possibility that $\theta \lt x_i.$ Some consequences are (1) convergence is faster than $O(n^{-1/2})$ and (2) the standardized asymptotic distribution of $\hat\theta$ is non-normal. $\endgroup$ – whuber Nov 13 '19 at 13:29
  • $\begingroup$ I guess it depends on what you mean as a "boundary point". In this context, I took this to mean the boundaries of the set of nonzero values of the likelihood function. Unless I'm mistaken, the likelihood is $L_\mathbf{x}(\theta) = \theta^{-n} \cdot \mathbb{I}(\theta \geqslant x_{(n)})$, so the MLE $\hat{\theta} = x_{(n)}$ does occur at a boundary point that is not a critical point of the function. $\endgroup$ – Reinstate Monica Nov 13 '19 at 13:49
  • $\begingroup$ Because that's not the usual meaning of a "boundary point" of a function, I felt it would be useful to provide a clarifying comment. Note, too, that $x_{n}$ is a critical point according to the standard definition: en.wikipedia.org/wiki/Critical_point_(mathematics). However, similar examples occur (for the same underlying reasons) even where the likelihood function is infinitely differentiable. The three-parameter Lognormal family provides a good example. $\endgroup$ – whuber Nov 13 '19 at 14:09
  • $\begingroup$ Sorry, I meant to say it is not a "stationary point" (edited post to clarify) since the derivative does not exist at this point. $\endgroup$ – Reinstate Monica Nov 13 '19 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.