Sample average of fixed-subsample MLE's Theory of MLE's state that, for $X_1, \cdots, X_n \overset{\text{i.i.d}}{\sim} F(\theta)$, that $\widehat{\theta}_{MLE}(\textbf{X}_n) \overset{p}{\rightarrow} \theta$ (assuming some regularity conditions).
Now suppose $X_{ij} \overset{\text{i.i.d}}{\sim} F(\theta)$ for all $i$ and $j = 1, \cdots, J$ for fixed $J$. Compute $\widehat{\theta}_{MLE}(\textbf{X}_{iJ})$ (that is, the MLE based on the subsample $i1, \cdots, iJ$). Then calculate $\widehat{\theta}_{MLE}^* = \frac{1}{n}\sum_{i=1}^{n}\widehat{\theta}_{MLE}(\textbf{X}_{iJ})$. Is it also true that $\widehat{\theta}_{MLE}^* \overset{p}{\rightarrow} \theta$? Are there conditions to guarantee this is true?
 A: There are two cases to consider. The first is that $J$ is fixed but $n \rightarrow \infty$. The second is $n$ is fixed and $J \rightarrow \infty$. 
In either case we have a collection of estimators $\hat \theta_i$ for $i=1, \dots, n$ computed on $(X_{i,1}, \dots, X_{i,J})$. $J$ getting bigger means that each individual estimator has more data; $n$ getting bigger means our final $\hat \theta^*$ averages more things.
Case 1: $J$ fixed, $n \rightarrow \infty$
Each $\hat \theta_i$ is iid so by the law of large numbers $\frac{1}{n} \sum_{i=1}^n \hat \theta_i \rightarrow_p \mu$ where $\mu = \mathbb E(\hat \theta_1)$. If each $\hat \theta_i$ is unbiased, then this converges in probability on the true parameter $\theta$. It is not guaranteed that the MLE computed on a finite sample is unbiased, so in the finite $J$ case we may not have $\hat \theta^* \rightarrow_p \theta$.
Case 2: $n$ fixed, $J \rightarrow \infty$
Now we are averaging a finite number of things, but each thing is converging on $\theta$ by the consistency of the MLE. By Slutsky's theorem we have $\hat \theta_1 + \hat \theta_2 \rightarrow_p 2\theta$, so we can proceed inductively to show that $\sum \limits_{i=1}^n \hat \theta_i \rightarrow_p n \theta$, and therefore $\frac{1}{n}\sum \limits_{i=1}^n \hat \theta_i \rightarrow_p \theta$.
The difference between these cases really comes down to bias vs. consistency. The MLE may be biased, so for a finite sample its expectation may not equal the target parameter, but asymptotically it converges on that parameter. This means that in the first case our $\hat \theta^*$ converges on the expectation of a finite sample MLE, while in case 2 $\hat \theta^*$ is a finite average of sequences, each of which converges on the true $\theta$, and therefore it does too.
As far as regularity conditions, beyond the $X_{ij}$ being iid, I think everything we needed here is inherited from the properties and regularity conditions of a single $\hat \theta_i$. The LLN requires a finite first moment, but I can't imagine the MLE would work as advertised if we don't have that.
