convergence of an average of consistent estimators? Let $\frac{1}{n}\sum_{i=1}^n X_i^j \overset{\text{p}}{\to} \mu^j$ for each $j$ (as $n \to \infty$ ). Under what conditions can we guarantee that
$$
\frac{1}{nm}\sum_{j=1}^m\sum_{i=1}^nX_i^j \overset{\text{p}}{\to} E[\mu]?
$$
Is there a way to guarantee it that doesn't involve assuming uniform convergence in probability? The sequences are uniformly convergent in probability if
$$
\sup_{j} \left| \frac{1}{n}\sum_{i=1}^n X_i^j - \mu^j \right| \overset{\text{p}}{\to} 0.
$$
 A: Assuming $E[\mu] = \frac{1}{m} \sum_{j=1}^m \mu^j$ and $m \rightarrow \infty$. If you're ok with passing to a subsequence $n_m$ so that 
$$
P( | \frac{1}{n_m}\sum_{i=1}^{n_m} X_{i}^j - \mu^j | < \frac{1}{m^2} ) > 1 - \frac{1}{m^2},
$$
then the result would hold for the subsequence (more precisely, sub-array).
In general, it is not true that
$$
\epsilon^j_n \stackrel{p}{\rightarrow} 0 \; \mbox{ as } n \rightarrow \infty, \;\; \forall j 
$$
implies 
$$
\frac{1}{m} \sum_{j = 1}^m \epsilon^j_n \stackrel{p}{\rightarrow} 0 \; \mbox{ as } n, \; m \rightarrow \infty.
$$
Indeed, it is not even true for deterministic sequences, which is a special case. Let
$$
\epsilon^j_n = j \cdot (\log (n+1) - \log(n)),
$$ 
then $\epsilon^j_n \rightarrow 0$ as $n \rightarrow \infty$ for all $j$.
Averaging across $j$ gives
$$
\frac{1}{m} \sum_{j = 1}^m \epsilon^j_n = O\left(m \cdot ( \log (n+1) - \log(n) ) \right)
$$
which does not converge to $0$ as $n, m \rightarrow 0$.
(You can easily cook up a sequence $X^j_n$ so that $\frac{1}{n}\sum_{i=1}^nX_n^j = \epsilon^j_n$, making it a counter-example to the claim.)
A: By the triangle-inequality
\begin{align*}
0 &\le \left|\frac{1}{nm}\sum_{j=1}^m\sum_{i=1}^nX_i^j - E[\mu]\right| \\
&\le
\left|\frac{1}{nm}\sum_{j=1}^m\sum_{i=1}^nX_i^j - m^{-1}\sum_{j=1}^m\mu^j \right| + \left|m^{-1}\sum_{j=1}^m\mu^j - E[\mu]\right| \\
&\le
\sup_{\mu \in \Theta}
\left|\frac{1}{n}\sum_{i=1}^n \bar{X}_\mu - m^{-1}\sum_{j=1}^m\mu^j \right| + \left|m^{-1}\sum_{j=1}^m\mu^j - E[\mu]\right|.
\end{align*}
With some nonstandard assumptions, you can invoke Theorem 2.1 of this to guarantee the first term converges to $0$. The second term converges to $0$ by the standard law of large numbers.
