3
$\begingroup$

Theory of MLE's state that, for $X_1, \cdots, X_n \overset{\text{i.i.d}}{\sim} F(\theta)$, that $\widehat{\theta}_{MLE}(\textbf{X}_n) \overset{p}{\rightarrow} \theta$ (assuming some regularity conditions).

Now suppose $X_{ij} \overset{\text{i.i.d}}{\sim} F(\theta)$ for all $i$ and $j = 1, \cdots, J$ for fixed $J$. Compute $\widehat{\theta}_{MLE}(\textbf{X}_{iJ})$ (that is, the MLE based on the subsample $i1, \cdots, iJ$). Then calculate $\widehat{\theta}_{MLE}^* = \frac{1}{n}\sum_{i=1}^{n}\widehat{\theta}_{MLE}(\textbf{X}_{iJ})$. Is it also true that $\widehat{\theta}_{MLE}^* \overset{p}{\rightarrow} \theta$? Are there conditions to guarantee this is true?

$\endgroup$

1 Answer 1

4
$\begingroup$

There are two cases to consider. The first is that $J$ is fixed but $n \rightarrow \infty$. The second is $n$ is fixed and $J \rightarrow \infty$.

In either case we have a collection of estimators $\hat \theta_i$ for $i=1, \dots, n$ computed on $(X_{i,1}, \dots, X_{i,J})$. $J$ getting bigger means that each individual estimator has more data; $n$ getting bigger means our final $\hat \theta^*$ averages more things.

Case 1: $J$ fixed, $n \rightarrow \infty$

Each $\hat \theta_i$ is iid so by the law of large numbers $\frac{1}{n} \sum_{i=1}^n \hat \theta_i \rightarrow_p \mu$ where $\mu = \mathbb E(\hat \theta_1)$. If each $\hat \theta_i$ is unbiased, then this converges in probability on the true parameter $\theta$. It is not guaranteed that the MLE computed on a finite sample is unbiased, so in the finite $J$ case we may not have $\hat \theta^* \rightarrow_p \theta$.

Case 2: $n$ fixed, $J \rightarrow \infty$

Now we are averaging a finite number of things, but each thing is converging on $\theta$ by the consistency of the MLE. By Slutsky's theorem we have $\hat \theta_1 + \hat \theta_2 \rightarrow_p 2\theta$, so we can proceed inductively to show that $\sum \limits_{i=1}^n \hat \theta_i \rightarrow_p n \theta$, and therefore $\frac{1}{n}\sum \limits_{i=1}^n \hat \theta_i \rightarrow_p \theta$.

The difference between these cases really comes down to bias vs. consistency. The MLE may be biased, so for a finite sample its expectation may not equal the target parameter, but asymptotically it converges on that parameter. This means that in the first case our $\hat \theta^*$ converges on the expectation of a finite sample MLE, while in case 2 $\hat \theta^*$ is a finite average of sequences, each of which converges on the true $\theta$, and therefore it does too.

As far as regularity conditions, beyond the $X_{ij}$ being iid, I think everything we needed here is inherited from the properties and regularity conditions of a single $\hat \theta_i$. The LLN requires a finite first moment, but I can't imagine the MLE would work as advertised if we don't have that.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.