Say that $X$ is a set $\{X_1, X_2, ..., X_N\}$ of (non-independent) random variables, and that $\hat{\mu}$ is a set $\{\hat{\mu}_1, \hat{\mu}_2, ..., \hat{\mu}_N\}$ of estimators. Each $\hat{\mu}_i$ is an estimator of $\mu_i$, the mean of the corresponding $X_i$.

Given that the estimators are consistent, and given that each estimator is calculated using $n_i$ samples, the central limit theorem gives a relationship between the variance of each estimator and its underlying random variable: \begin{equation} V[\hat{\mu}_i] \approx \frac{V[X_i]}{n_i} \end{equation}

where the above approximation becomes more accurate the larger the sample size $n_i$ is.

Is there an equivalent relationship between the covariances? My gut tells me it might be: \begin{equation} Cov[\hat{\mu}_i, \hat{\mu}_j] \approx \frac{Cov[X_i, X_j]}{\sqrt{n_in_j}} \end{equation}

but that's just a guess that I don't how to dis/prove.

Is there any relationship of this sort between the covariances of the estimator and the underlying? I'm looking for proofs and/or pointers to a definitive reference.


I'm working with a type of simulation that can be thought of as consisting of $N$ consecutive phases. Each phase $i$ is associated with a random variable $X_i$, and the goal of each phase can be thought of estimating the mean $\mu_i$ of $X_i$. During each phase $i$, $n_i$ samples $X_{ij}$ are drawn from $X_i$, and then an estimate is calculated as $\hat{\mu_i} = \frac{1}{n_i}\sum_{j=1}^{n_i} X_{ij}$ (so the estimator $\hat{\mu_i}$ is a sample mean). Once phase $i$ is finished, phase $i + 1$ is then run, and the process is repeated until all phases have finished.

The output value of interest from this type of simulation is the product of the sample means: \begin{equation} f(\hat{\mu})=\prod_{i=1}^N\hat{\mu_i} \end{equation} What I really want to know is how to calculate the variance $V[f(\hat{\mu})]$ as a function of the sample counts $n_i$. In the limit of large sample counts, there's a relatively simple formula for the variance of the product: \begin{equation} V[f(\hat{\mu})] \approx \prod_{i}\mu_i^2 (\sum_i\frac{V[\hat{\mu_i}]}{\mu_i^2} + \sum_{i\neq j} Cov[\hat{\mu_i}, \hat{\mu_j}]) \end{equation}

The $V[\hat{\mu_i}]$ terms can be rewritten using the first equation above: \begin{equation} V[f(\hat{\mu})] \approx \prod_{i}\mu_i^2 (\sum_i\frac{V[X_i]}{\mu_i^2 n_i} + \sum_{i\neq j} Cov[\hat{\mu_i}, \hat{\mu_j}]) \end{equation}

So all I need now is a similar way to rewrite the $Cov[\hat{\mu_i}, \hat{\mu_j}]$ terms.

  • $\begingroup$ What does $E[x_i] = X_i$ mean? $\endgroup$
    – user158565
    Commented Oct 24, 2018 at 1:17
  • $\begingroup$ @a_statistician I fixed up the lead-in and deleted the offending statement. Does it look right to you now? $\endgroup$
    – tel
    Commented Oct 24, 2018 at 2:27
  • $\begingroup$ You have not specified the form of your estimators, so it is not really possible to specify their variance, covariance, etc. (And the CLT is not generally required to find the variance of an estimator.) Do you intend for your estimators to be sample means of some set of variables, and if so, of what variables? $\endgroup$
    – Ben
    Commented Oct 24, 2018 at 4:50
  • $\begingroup$ @Ben I'm trying to get the most general answer possible, so I was being a little vague on purpose. But, all of the estimators that I'm working with are indeed sample means, so you can make that assumption if it's needed. I'll add some more concrete details about the whole setup to the context section $\endgroup$
    – tel
    Commented Oct 24, 2018 at 4:58
  • $\begingroup$ @tel: Given that you have only specified one random variable of each kind, what are they sample means of? Can we assume that you are actually observing multiple instantiations of each of the random variables in your set? $\endgroup$
    – Ben
    Commented Oct 24, 2018 at 5:20

1 Answer 1


In fact, decrease your $N$ to 2. What you say is you have data $X_{1i}, i= 1 \text{ to } n_1$ and $X_{2i}, i= 1 \text{ to } n_2$. Here $X_{1i}$ and $X_{2i}$ are dependent, but $X_{1i}$ and $X_{1j}$ are independent. Same as $X_{2j}$. You want to know $Cov(\frac 1{n_1}\sum_{i=1}^{n_1}X_{1i},\frac 1{n_2}\sum_{i=1}^{n_2}X_{2i})$.

Suppose $n_1>n_2$, i.e., $n_1=n_2+k$, then $(X_{1(n2+1)},...,X_{1(n_2+k)})$ are independent with $X_{1i}$ and $X_{2i}$ for $i=1,...,n_2$. Then we have $$Cov\left(\frac 1{n_1}\sum_{i=1}^{n_1}X_{1i},\frac 1{n_2}\sum_{i=1}^{n_2}X_{2i}\right)= Cov\left(\frac 1{n_1}\sum_{i=1}^{n_2}X_{i1},\frac 1{n_2}\sum_{i=1}^{n_2}X_{2i}\right)$$ $$= \frac 1 {n_1n_2}\sum_{i=1}^{n_2}Cov(X_{1i},X_{2i}) = \frac 1{n_1}Cov(X_{1i},X_{2i})$$


Based on "Every sample drawn from a single Xi are uncorrelated with each other, but you should assume that every sample from Xi is correlated with every sample from Xj, for every j≠i. I've also fixed up the notation and added more details in the question.", here is a general method:

Let $$Y=(X_{11},X_{12},...,X_{1n_1},X_{21},X_{22},...,X_{2n_2},...,X_{N1},X_{N2},...,X_{Nn_N})^t $$ $$ A_{(N\times M)}=\left(\matrix{1/n_1 & 1/n_1 &...& 1/n_1 & 0 & 0 &...\\ 0&0&...& 1/n_2&...&1/n_2&0&...&0\\ ...&...&...&...\\ 0&...&0& 1/n_N& 1/n_N &...& 1/n_N }\right)$$ where $M = \sum_{j=1}^Nn_j$.

Then $$(\hat μ_1,\hat μ_2,...,\hat μ_N)^t=AY$$
$$\mathrm{Var}(\hat μ_1,\hat μ_2,...,\hat μ_N) = A\mathrm{Cov}(Y)A^t $$ So after specifying $\mathrm{Cov}(Y)$, everything is easy. But only you can can do this, because others do not know your real situation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.