I want to estimate the mean of a function f, i.e. $$E_{X,Y}[f(X,Y)]$$ where $X$ and $Y$ are independent random variables. I have samples of f but not iid: There are iid samples for $Y_1,Y_2,\dots Y_n$ and for each $Y_i$ there are $n_i$ samples from $X$: $X_{i,1},X_{i,2},\dots, X_{i,n_i}$

So in total I have samples $f(X_{1,1},Y_1) \dots f(X_{1,n_1},Y_1 ) \dots f(X_{i,j},Y_i) \dots f(X_{n,n_n},Y_n)$

To estimate the mean I compute $$\mu=\sum_{i=1}^n 1/n * \sum_{j=1}^{n_i}\frac{ f(X_{i,j},Y_i)}{n_i}$$ Obviously $$E_{X,Y}[\mu]=E_{X,Y}[f(X,Y)]$$ so $\mu$ is an unbiased estimator. I am wondering now what $Var(\mu)$, i.e. the variance of the estimator is.

Edit 2: Is this the correct variance? $$Var(\mu)=\frac{Var_Y(\mu_i)}{n}+\sum_{i=1}^n \frac{Var_X(f(X,Y_i)))}{n_i*n^2}$$ It seems to work in the limit, i.e. if n=1 and all $n_i=\infty$ the variance just becomes the variance of the means. And if $n_i=1$ the formula becomes the standard formula for the variance of estimators. Is this correct? How can I proof that it is?

Edit (Ignore this):

So I think I made some progress: Let us first define $\mu_i=\sum_{j=1}^{n_i}\frac{ f(X_{i,j},Y_i)}{n_i}$ which is an unbiased estimator of $E_X[f(X,Y_i)]$.

Using the standard formula for variance we can write:

$$Var(\mu)=1/n^2 \sum_{l=1}^n \sum_{k=1}^n Cov(\mu_l,\mu_k)$$ This can be simplified to $$1/n^2( \sum_{i=1}^n Var(\mu_l)+ 1/n^2\sum_{l=1}^n \sum_{k=l+1}^n 2*Cov(\mu_l,\mu_k))$$ and because the $X_{ij}$s are drawn independently we can further simplify this to $$1/n^2( \sum_{i=1}^n 1/n_i Var(f(X_{i,j},Y_i))+1/n^2 \sum_{l=1}^n \sum_{k=l+1}^n 2*Cov(\mu_l,\mu_k))$$ And for the covariance: \begin{align} Cov(\mu_l,\mu_k)&=Cov(\sum_{j=1}^{n_l} \frac{f(X_{j,l},Y_l)}{n_{l}},\sum_{j=1}^{n_k} \frac{f(X_{j,k},Y_k)}{n_{k}})\\ &=\frac{1}{(n_k*n_l)}*Cov(\sum_{j=1}^{n_l} f(X_{j,l},Y_l),\sum_{j=1}^{n_k} f(X_{j,k},Y_k))\\ &=\frac{1}{(n_k*n_l)}*\sum_{j=1}^{n_l}\sum_{j=1}^{n_k}Cov( f(X,Y_l), f(X,Y_k))\\ &=\frac{n_k*n_l}{(n_k*n_l)}Cov( f(X_{i,l},Y_l), f(X_{i,k},Y_k))\\ &=Cov( f(X,Y_l), f(X,Y_k)) \end{align} So plugging this back in we get $$1/n^2( \sum_{i=1}^n 1/n_i Var(f(X,Y_i))+1/n^2 \sum_{l=1}^n \sum_{k=l+1}^n 2*Cov(f(X,Y_l),f(X,Y_k)))$$ I have multiple questions now:

  1. Is the calculation above correct?

  2. How can I estimate $Cov(f(X,Y_l),f(X,Y_k)))$ from the samples given?

  3. Does the variance converge to 0 if I let n go to infinity?


1 Answer 1


Q1: No, it's not quite right. You omit the subscripts in line 3 of your final derivation of the covariance. That obscures the fact that the two RVs labeled "X" are in fact independent of one another: one had an $\ell$ subscript and the other a $k$. In that whole block of equalities, the only nonzero terms should be when $k=\ell$, because functions of independent inputs are independent. (I assume you are okay with saying $X_{12}, Y_1$ is independent of $X_{22}, Y_2$ even though this does not follow, strictly speaking, from pairwise claims of independence among all the $X$'s and $Y$'s.)

Q2: From above, that term is nonzero only when $k=\ell$, and in that case, it reduces to $Cov(f(X_{jk}, Y_k), f(X_{jk}, Y_k)) = Var(f(X_{jk}, Y_k))$. The result after the sum is $Cov(\mu_k, \mu_k) = \frac{1}{n_k} Var(f(X_{jk}, Y_k))$.

Q3: Yes: after these modifications, you will have only a linear number of terms in the very last sum, so the denominator's quadratic term will win.

  • $\begingroup$ The answer to "Does the variance converge to 0 if I let n go to infinity?" is "Yes". $\endgroup$ Commented Nov 8, 2015 at 6:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.