# Computing the Variance of an MLE

Suppose we have i.i.d. $n$ observations $(X_1,X_2,...X_n)$ from a population with density $$f_\theta(x)=\begin{cases}\theta x^{\theta-1}&\text{ if }0\leq x\leq 1\\0&\text{otherwise.}\end{cases}$$ Note that $\theta>0$. If $T_n$ denotes the maximum likelihood estimator of $\theta$ given that $n$ is the sample size, then show that $$\text{var}(T_n)\stackrel{n\to\infty}{\longrightarrow} 0$$

This is the problem I am trying to solve: I computed $$T_n=\dfrac{-n}{\sum_{i=1}^{n}\ln(X_i)}$$ But I am stuck in finding $\text{var}(T_n)$ and showing that $\text{var}(T_n)\to0$ as $n\to\infty$.

• can you relate $T_n$ to an empirical average? – Xi'an Jan 17 '15 at 13:21
• I don't quite get your question. I have observed that $T_n$ is the inverse of the A.M. of $\ln(X_i)$ which is trivial to see, along with a negative sign. But we are required to compute the variance of this M.L.E. That's reverting back to population parameters, as I am not interested in the "sample" variance. – Landon Carter Jan 17 '15 at 13:24
• I was suggesting this approach to apply the delta method and deduce from the CLT that the variance of $T_n$ goes to zero. – Xi'an Jan 17 '15 at 13:29
• That seems interesting. So now I need to find a suitable function $g$ to apply the delta method such that $(g(X)-g(\mu))\sqrt{n}\to N(0,\sigma^2(g'(x))^2)$. – Landon Carter Jan 17 '15 at 13:34
• The transform is straightforward since you start from $\overline{\log(X_i)}$, i.e., $g(y)=1/y$. – Xi'an Jan 17 '15 at 13:39

$$1/T_n = \frac 1n \sum_{i=1}^n (-\ln X_i)$$

But $-\ln X_i \sim {\rm Exp}(1/\theta)$ (where $1/\theta$ is the scale parameter), which essentially is a Gamma distribution with shape parameter $1$, and so by the summation properties of the Gamma distribution,

$$\sum_{i=1}^n (-\ln X_i) \sim {\rm Gamma}(n, 1/\theta)$$.

Using the scaling properties of the Gamma distribution we obtain

$$1/T_n \sim {\rm Gamma}(n, 1/(n\theta))$$

Then the inverse of $1/T_n$ (i.e. $T_n$) follows an Inverse Gamma distribution, with same shape paramater and reciprocal scale parameter

$$T_n \sim {\rm InvGamma}(n, n\theta)$$

that has variance

$${\rm Var} (T_n) = \frac {n^2\theta^2}{(n-1)^2(n-2)}$$

as the Mathematica software gave.

The leading term in the numerator is $n^2$ while the leading term in the denominator is $n^3$ so the limit of the variance of $T_n$ with respect to $n$ goes to zero.

• I think this perhaps goes a fair bit beyond "hints and guidance" suitable for a fresh self-study question. – Glen_b Jan 18 '15 at 5:55
• @Glen_b You are right. I got carried away when I saw another answer providing the result through computation, and though it would be good to have also its analytical dimension, forgetting the self-study tag in the process. – Alecos Papadopoulos Jan 18 '15 at 16:01
• Never mind. In fact, I know nothing about an Inverse Gamma distribution. So had you not written the complete answer, I would have been in the dark. – Landon Carter Jan 18 '15 at 16:09
• The problem with using notation such as InverseGamma(a,b) is that there is no standard definition for what an InverseGamma(a,b) distribution defines. Some authors use IG(a,b), others use IG(a,1/b), and others still other notation ... What is the variance of an undefined object?? Using a lookup table can unfortunately yield the incorrect answer, if the text or lookup table happens to use different defns. Wiki sometimes uses one parameterisation for the pdf, and an inconsistent defn for moments etc (not here) - and they clash (not here). It is best to avoid the ambiguity by defining the pdf. – wolfies Jan 18 '15 at 17:58
• @wolfies This is the same problem we face with the Gamma distribution itself (and with the Exponential too). I have found that mentioning "shape-scale" or "shape-rate" parametrization, appears to resolve the communication problem. – Alecos Papadopoulos Jan 18 '15 at 20:15

Did you try to compute the Fisher information? Get the second derivative of the Log Likelihood, multiply by -1, and take the reciprocal. Evaluate at the MLE . Asymptotically, this approximates the variance of the MLE.

• I see that $$\hat\theta_{MLE}\sim AN(\theta,\theta^2/n)$$ Does this imply $E(\hat\theta)\to\theta$ and $\text{Var}(\hat\theta)\to \frac{\theta^2}{n}\to 0$ for large $n$? – StubbornAtom Aug 12 '18 at 19:02
• Reciprocal of Fisher information is indeed $\theta^2/n$. – StubbornAtom Aug 12 '18 at 19:10

The Problem

Let $(X_1, \dots, X_n)$ denote a random sample of size $n$ drawn on $$X \sim \text{PowerFunction}(\theta,1) \quad \text{ with pdf} \quad f(x) = \theta x^{\theta-1} \text{ where } \quad 0<x<1.$$ Let $Z = -\sum_{i=1}^{n}\ln(X_i). \quad$ Find: $\quad Var\big[\dfrac{n}{Z}\big]$

Solution

Let $Y = -ln(X)$. Then, by any of the usual methods of transformation, $Y \sim \text{Exponential}(\theta)$ with pdf $\theta e^{-\theta y}$. Next, the sum of $n$ iid Exponentials with rate $\theta$ is well-known to be $\text{Gamma}(n, 1/\theta)$ so $Z \sim \text{Gamma}(n, 1/\theta)$ with pdf, say, $g(z)$:

We seek $Var(\frac{n}{Z})$:

where I am using the Var function from the mathStatica package for Mathematica to automate the calculation.

The limit of the latter tends to 0, as $n \rightarrow \infty$, as required.

• $X$ is simply $\text{Beta}(\theta,1)$. – StubbornAtom Aug 12 '18 at 17:54