# Dependence of estimator covariance on sample count

Say that $$X$$ is a set $$\{X_1, X_2, ..., X_N\}$$ of (non-independent) random variables, and that $$\hat{\mu}$$ is a set $$\{\hat{\mu}_1, \hat{\mu}_2, ..., \hat{\mu}_N\}$$ of estimators. Each $$\hat{\mu}_i$$ is an estimator of $$\mu_i$$, the mean of the corresponding $$X_i$$.

Given that the estimators are consistent, and given that each estimator is calculated using $$n_i$$ samples, the central limit theorem gives a relationship between the variance of each estimator and its underlying random variable: $$\begin{equation} V[\hat{\mu}_i] \approx \frac{V[X_i]}{n_i} \end{equation}$$

where the above approximation becomes more accurate the larger the sample size $$n_i$$ is.

Is there an equivalent relationship between the covariances? My gut tells me it might be: $$\begin{equation} Cov[\hat{\mu}_i, \hat{\mu}_j] \approx \frac{Cov[X_i, X_j]}{\sqrt{n_in_j}} \end{equation}$$

but that's just a guess that I don't how to dis/prove.

Is there any relationship of this sort between the covariances of the estimator and the underlying? I'm looking for proofs and/or pointers to a definitive reference.

## Context

I'm working with a type of simulation that can be thought of as consisting of $$N$$ consecutive phases. Each phase $$i$$ is associated with a random variable $$X_i$$, and the goal of each phase can be thought of estimating the mean $$\mu_i$$ of $$X_i$$. During each phase $$i$$, $$n_i$$ samples $$X_{ij}$$ are drawn from $$X_i$$, and then an estimate is calculated as $$\hat{\mu_i} = \frac{1}{n_i}\sum_{j=1}^{n_i} X_{ij}$$ (so the estimator $$\hat{\mu_i}$$ is a sample mean). Once phase $$i$$ is finished, phase $$i + 1$$ is then run, and the process is repeated until all phases have finished.

The output value of interest from this type of simulation is the product of the sample means: $$\begin{equation} f(\hat{\mu})=\prod_{i=1}^N\hat{\mu_i} \end{equation}$$ What I really want to know is how to calculate the variance $$V[f(\hat{\mu})]$$ as a function of the sample counts $$n_i$$. In the limit of large sample counts, there's a relatively simple formula for the variance of the product: $$\begin{equation} V[f(\hat{\mu})] \approx \prod_{i}\mu_i^2 (\sum_i\frac{V[\hat{\mu_i}]}{\mu_i^2} + \sum_{i\neq j} Cov[\hat{\mu_i}, \hat{\mu_j}]) \end{equation}$$

The $$V[\hat{\mu_i}]$$ terms can be rewritten using the first equation above: $$\begin{equation} V[f(\hat{\mu})] \approx \prod_{i}\mu_i^2 (\sum_i\frac{V[X_i]}{\mu_i^2 n_i} + \sum_{i\neq j} Cov[\hat{\mu_i}, \hat{\mu_j}]) \end{equation}$$

So all I need now is a similar way to rewrite the $$Cov[\hat{\mu_i}, \hat{\mu_j}]$$ terms.

• What does $E[x_i] = X_i$ mean? – user158565 Oct 24 '18 at 1:17
• @a_statistician I fixed up the lead-in and deleted the offending statement. Does it look right to you now? – tel Oct 24 '18 at 2:27
• You have not specified the form of your estimators, so it is not really possible to specify their variance, covariance, etc. (And the CLT is not generally required to find the variance of an estimator.) Do you intend for your estimators to be sample means of some set of variables, and if so, of what variables? – Ben Oct 24 '18 at 4:50
• @Ben I'm trying to get the most general answer possible, so I was being a little vague on purpose. But, all of the estimators that I'm working with are indeed sample means, so you can make that assumption if it's needed. I'll add some more concrete details about the whole setup to the context section – tel Oct 24 '18 at 4:58
• @tel: Given that you have only specified one random variable of each kind, what are they sample means of? Can we assume that you are actually observing multiple instantiations of each of the random variables in your set? – Ben Oct 24 '18 at 5:20

## 1 Answer

In fact, decrease your $$N$$ to 2. What you say is you have data $$X_{1i}, i= 1 \text{ to } n_1$$ and $$X_{2i}, i= 1 \text{ to } n_2$$. Here $$X_{1i}$$ and $$X_{2i}$$ are dependent, but $$X_{1i}$$ and $$X_{1j}$$ are independent. Same as $$X_{2j}$$. You want to know $$Cov(\frac 1{n_1}\sum_{i=1}^{n_1}X_{1i},\frac 1{n_2}\sum_{i=1}^{n_2}X_{2i})$$.

Suppose $$n_1>n_2$$, i.e., $$n_1=n_2+k$$, then $$(X_{1(n2+1)},...,X_{1(n_2+k)})$$ are independent with $$X_{1i}$$ and $$X_{2i}$$ for $$i=1,...,n_2$$. Then we have $$Cov\left(\frac 1{n_1}\sum_{i=1}^{n_1}X_{1i},\frac 1{n_2}\sum_{i=1}^{n_2}X_{2i}\right)= Cov\left(\frac 1{n_1}\sum_{i=1}^{n_2}X_{i1},\frac 1{n_2}\sum_{i=1}^{n_2}X_{2i}\right)$$ $$= \frac 1 {n_1n_2}\sum_{i=1}^{n_2}Cov(X_{1i},X_{2i}) = \frac 1{n_1}Cov(X_{1i},X_{2i})$$

UPDATE

Based on "Every sample drawn from a single Xi are uncorrelated with each other, but you should assume that every sample from Xi is correlated with every sample from Xj, for every j≠i. I've also fixed up the notation and added more details in the question.", here is a general method:

Let $$Y=(X_{11},X_{12},...,X_{1n_1},X_{21},X_{22},...,X_{2n_2},...,X_{N1},X_{N2},...,X_{Nn_N})^t$$ $$A_{(N\times M)}=\left(\matrix{1/n_1 & 1/n_1 &...& 1/n_1 & 0 & 0 &...\\ 0&0&...& 1/n_2&...&1/n_2&0&...&0\\ ...&...&...&...\\ 0&...&0& 1/n_N& 1/n_N &...& 1/n_N }\right)$$ where $$M = \sum_{j=1}^Nn_j$$.

Then $$(\hat μ_1,\hat μ_2,...,\hat μ_N)^t=AY$$
$$\mathrm{Var}(\hat μ_1,\hat μ_2,...,\hat μ_N) = A\mathrm{Cov}(Y)A^t$$ So after specifying $$\mathrm{Cov}(Y)$$, everything is easy. But only you can can do this, because others do not know your real situation.