Dependence of estimator covariance on sample count

Question

Say that $X$ is a set $\{X_1, X_2, \ldots, X_N\}$ of (non-independent) random variables, and that $\hat{\mu}$ is a set $\{\hat{\mu}_1, \hat{\mu}_2, \ldots, \hat{\mu}_N\}$ of estimators. Each $\hat{\mu}_i$ is an estimator of $\mu_i$, the mean of the corresponding $X_i$.

Given that the estimators are consistent, and given that each estimator is calculated using $n_i$ samples, the central limit theorem gives a relationship between the variance of each estimator and its underlying random variable: \begin{equation} V[\hat{\mu}_i] \approx \frac{V[X_i]}{n_i} \end{equation}

where the above approximation becomes more accurate the larger the sample size $n_i$ is.

Is there an equivalent relationship between the covariances? My gut tells me it might be: $$\operatorname{ Cov}[\hat{\mu}_i, \hat{\mu}_j] \approx \frac{\operatorname {Cov}[X_i, X_j]}{\sqrt{n_in_j}} $$

but that's just a guess that I don't how to dis/prove.

Is there any relationship of this sort between the covariances of the estimator and the underlying? I'm looking for proofs and/or pointers to a definitive reference.

Context

I'm working with a type of simulation that can be thought of as consisting of $N$ consecutive phases. Each phase $i$ is associated with a random variable $X_i$, and the goal of each phase can be thought of estimating the mean $\mu_i$ of $X_i$. During each phase $i$, $n_i$ samples $X_{ij}$ are drawn from $X_i$, and then an estimate is calculated as $\hat{\mu_i} = \frac{1}{n_i}\sum_{j=1}^{n_i} X_{ij}$ (so the estimator $\hat{\mu_i}$ is a sample mean). Once phase $i$ is finished, phase $i + 1$ is then run, and the process is repeated until all phases have finished.

The output value of interest from this type of simulation is the product of the sample means:

$$ f(\hat{\mu})=\prod_{i=1}^N\hat{\mu_i} $$

What I really want to know is how to calculate the variance $V[f(\hat{\mu})]$ as a function of the sample counts $n_i$. In the limit of large sample counts, there's a relatively simple formula for the variance of the product:

$$ V[f(\hat{\mu})] \approx \prod_{i}\mu_i^2 \left(\sum_i\frac{V[\hat{\mu_i}]}{\mu_i^2} + \sum_{i\neq j} \operatorname{Cov}[\hat{\mu_i}, \hat{\mu_j}]\right) $$

The $V[\hat{\mu_i}]$ terms can be rewritten using the first equation above: $$ V[f(\hat{\mu})] \approx \prod_{i}\mu_i^2 \left(\sum_i\frac{V[X_i]}{\mu_i^2 n_i} + \sum_{i\neq j} \operatorname{Cov}[\hat{\mu_i}, \hat{\mu_j}]\right) $$

So all I need now is a similar way to rewrite the $\operatorname {Cov}[\hat{\mu_i}, \hat{\mu_j}]$ terms.

Answer 1

In fact, decrease your $N$ to 2. What you say is you have data $X_{1i}, i= 1 \text{ to } n_1$ and $X_{2i}, i= 1 \text{ to } n_2$. Here $X_{1i}$ and $X_{2i}$ are dependent, but $X_{1i}$ and $X_{1j}$ are independent. Same as $X_{2j}$. You want to know $Cov(\frac 1{n_1}\sum_{i=1}^{n_1}X_{1i},\frac 1{n_2}\sum_{i=1}^{n_2}X_{2i})$.

Suppose $n_1>n_2$, i.e., $n_1=n_2+k$, then $(X_{1(n2+1)},...,X_{1(n_2+k)})$ are independent with $X_{1i}$ and $X_{2i}$ for $i=1,...,n_2$. Then we have $$\operatorname{Cov}\left(\frac 1{n_1}\sum_{i=1}^{n_1}X_{1i},\frac 1{n_2}\sum_{i=1}^{n_2}X_{2i}\right)= \operatorname{Cov}\left(\frac 1{n_1}\sum_{i=1}^{n_2}X_{i1},\frac 1{n_2}\sum_{i=1}^{n_2}X_{2i}\right)$$ $$= \frac 1 {n_1n_2}\sum_{i=1}^{n_2}\operatorname{Cov}(X_{1i},X_{2i}) = \frac 1{n_1}\operatorname{Cov}(X_{1i},X_{2i})$$

UPDATE

Based on "Every sample drawn from a single Xi are uncorrelated with each other, but you should assume that every sample from Xi is correlated with every sample from Xj, for every j≠i. I've also fixed up the notation and added more details in the question.", here is a general method:

Let $$Y=(X_{11},X_{12},...,X_{1n_1},X_{21},X_{22},...,X_{2n_2},...,X_{N1},X_{N2},...,X_{Nn_N})^t $$ $$ A_{(N\times M)}=\left(\matrix{1/n_1 & 1/n_1 &...& 1/n_1 & 0 & 0 &...\\ 0&0&...& 1/n_2&...&1/n_2&0&...&0\\ ...&...&...&...\\ 0&...&0& 1/n_N& 1/n_N &...& 1/n_N }\right)$$ where $M = \sum_{j=1}^Nn_j$.

Then $$(\hat \mu_1,\hat \mu_2,...,\hat \mu_N)^t=AY$$
$$\mathrm{Var}(\hat \mu_1,\hat \mu_2,...,\hat \mu_N) = A\mathrm{Cov}(Y)A^t $$ So after specifying $\mathrm{Cov}(Y)$, everything is easy. But only you can can do this, because others do not know your real situation.