0
$\begingroup$

Suppose I have data $\{X_i\}_{i=1}^{n}\sim \text{Uniform}[0,\theta_0]$. We know that the maximum likelihood estimator of $\theta_0$ is $\widehat{\theta}=\max\{X_1,...,X_n\}$. How to show that $\widehat{\theta}$ converges to $\theta_0$ at rate $n^{-1}$? Thanks!

$\endgroup$
1
$\begingroup$

When $(X_i)$ is a sequence of uniformly distributed variables $\mathcal{U}_{[0,\theta_0]}$ we can derive the distribution of the MLE $\hat \theta = \max X_i$.
We will show that $n( \theta_0 - \hat \theta)$ converge towards a non trivial (i.e. not $0$) distribution, which means that the rate of convergence of the MLE is $n^{-1}$.

First note that since $\max X_i \leq \theta_0$, $n( \theta_0 - \hat \theta) \geq 0$.

Then, for $t \geq 0$,

\begin{align*} \mathbb P\left( n(\theta_0 -\hat \theta) \leq t \right) &=1- \mathbb P\left( \hat \theta \leq \theta_0 - \frac{t}{n} \right) \\ &=1-\mathbb P\left( \max X_i \leq \theta_0 - \frac{t}{n} \right) \\ \end{align*}

Since $\max X_i \leq \theta_0 - \frac{t}{n} \iff \forall i, X_i \leq \theta_0 - \frac{t}{n}$ and since all $X_i$ are independent and identically distributed, \begin{align*} \mathbb P\left( \max X_i \leq \theta_0 - \frac{t}{n} \right) &=\prod_{i=1}^n\mathbb P\left( X_i \leq \theta_0 - \frac{t}{n} \right) \\ &=\mathbb P\left( X_1 \leq \theta_0 - \frac{t}{n} \right)^n \end{align*}

For $n$ large enough we have $\frac{t}{n} \leq \theta_0$ and then, \begin{align*} \mathbb P\left( X_1 \leq \theta_0 - \frac{t}{n} \right)^n &= \left(\frac{\theta_0 - \frac{t}{n} }{\theta_0} \right)^n \\ &=\left(1 - \frac{t}{\theta_0 n}\right)^n \end{align*}

As $n \to \infty$ the last line is equivalent to $\exp\left(-\frac{t}{\theta_0}\right).$

So for $t \geq 0$: $$ \mathbb P\left( n(\theta_0 -\hat \theta) \leq t \right) \xrightarrow[n \to \infty]{} 1- \exp\left(-\frac{t}{\theta_0}\right). $$

We can see that $1-\exp\left(-\frac{t}{\theta_0}\right)$ is the cumulative distribution function of an exponential random variable with rate $\frac{1}{\theta_0}$.

Thus, $n (\theta_0 - \hat \theta )$ is asymptotically exponentially distributed, and the rate of convergence of $\hat \theta$ is $n^{-1}$.

Here is a R code to illustrate this result (with $\theta_0 = 3$ and $n=50$):

theta0<-3
n<-50
s<-sapply(1:1e4,function(i){
      X<-runif(n,0,theta0)
      return(n*(theta0-max(X)))
})
P_emp<-function(t){
      return(mean(s<=t))
}
#P_emp is an empirical cumulative distribution of n*(theta0 - MLE)
#P_emp should be close to the CDF of an exponential distribution

x<-seq(0,20,length.out = 1000)
y1<-sapply(x,function(t) P_emp(t))

plot(x,y1,type='l',col="red")
lines(x,pexp(x,rate=1/theta0),type='l',col="black")
legend(x=0,y=1,legend = c("Empirical CDF","Exponential 
CDF"),lty=1,col=c("red","black"))
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks! This is completely solved my problem! If I change the distribution to other distributions on $[0,\theta_0]$ and want to estimate the boundary ($\theta_0$) using $\underset{i}{max}\{X_i\}$, does the rate $n^{-1}$ still hold? $\endgroup$ – T34driver Jul 11 at 6:40
  • 2
    $\begingroup$ @JTS365 No I don't think it works for any distribution over $[0, \theta_0]$. Take for example $\theta_0 =1$ and the distribution with density $f(x) = 2-2x$ over $[0,1]$, then $\forall t$, $\mathbb P( n(1- \max X_i) \leq t) \to 0$ which means that $n(1-\max X_i) \to \infty$. In that case the rate is $n^{-\frac{1}{2}}$ $\endgroup$ – winperikle Jul 11 at 14:34
  • $\begingroup$ Thanks! Does the result generalize to bivariate uniform distribution on a rectangle $[0,\theta_0]\times [0,\theta_1]$? $\endgroup$ – T34driver Jul 16 at 23:41
  • $\begingroup$ @JTS365 Yes it does, if $(X_0,X_1) \sim U_{[0,\theta_0]\times [0,\theta1]}$ then $X_0 \perp X_1$ and $X_i \sim \mathcal U_{[0,\theta_i]}$ thus $\hat \theta_0 = \max X_{0,i}$ and $\hat \theta_1 = \max X_{1,i}$ and the rate for both estimators is $n^{-1}$. $\endgroup$ – winperikle Jul 17 at 22:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.