# Expectation of product of sample averages

I have a bunch of iid random variables $$X_i\sim q$$ and I have defined other random variables $$A_i = a(X_i)$$ and $$B_i = b(X_i)$$. Then I bumped into the following expression \begin{align} \mathbb{E}\left[\left(\frac{1}{N}\sum_{i=1}^N A_i\right)\left(\frac{1}{N}\sum_{i=1}^N B_i\right)\right] &= \frac{1}{N^2}\mathbb{E}\left[\sum_{i=1}^N \sum_{j=1}^N A_i B_j\right] \\ &= \frac{1}{N^2}\sum_{i=1}^N \sum_{j=1}^N \mathbb{E}[A_i B_j] \end{align}

What does $$\mathbb{E}[A_i B_j]$$ mean here?

It seems like a stupid question, but it's not clear to me what it represents. I tried to write it out as an integral but without success. I don't understand what distribution it is with respect to. Is it this? $$\mathbb{E}[A_i B_j] = \int\int a(x) b(y) q(dx) q(dy).$$ Surely, this cannot be correct, otherwise it would mean $$\mathbb{E}[A_i B_j] = \mathbb{E}[A]\mathbb{E}[B]$$, which would lead to $$\mathbb{E}[\bar{A}\bar{B}] = \mathbb{E}[A]\mathbb{E}[B],$$ where $$\bar{A}$$ and $$\bar{B}$$ are the sample averages above, and this cannot be true.

Given a probability space $$(\Omega, \mathcal{F}, \mathbb{P})$$ and a measurable space $$(E, \mathcal{E})$$ a random variable is a measurable function $$X:\Omega\to E$$. Expectations of this random variables are $$\mathbb{E}[X] = \int_{\Omega} X(\omega) \mathbb{P}(d\omega).$$ Given another random variable $$Y$$ on $$(\Omega, \mathcal{F}, \mathbb{P})$$ we know that $$\mathbb{E}[XY]\neq \mathbb{E}[X]\mathbb{E}[Y]$$ except when $$X$$ and $$Y$$ are independent. However, I don't see how $$\mathbb{E}[XY]$$ can ever be different from $$\mathbb{E}[X]\mathbb{E}[Y]$$. Indeed I could define $$Z(\omega, \omega') = X(\omega) Y(\omega')$$ as a random variable on $$(\Omega\times\Omega, \mathcal{F}\otimes\mathcal{F}, \mathbb{P}\otimes \mathbb{P})$$ and its expectation would be defined as \begin{align} \mathbb{E}[Z] &= \int_{\Omega\times\Omega} Z(\omega, \omega') \mathbb{P}(d\omega)\mathbb{P}(d\omega') \\ &= \int_\Omega \int_\Omega X(\omega) Y(\omega') \mathbb{P}(d\omega)\mathbb{P}(d\omega') \\ &= \left[\int_\Omega X(\omega) \mathbb{P}(d\omega)\right]\left[\int_\Omega Y(\omega') \mathbb{P}(d\omega')\right] \\ &= \mathbb{E}[X]\mathbb{E}[Y]. \end{align} Now, in the context of my main question, what would $$\mathbb{E}[A_i B_j]$$ be? I want to write it down as an integral but I am struggling to do so. Especially because $$a(X_i)$$ and $$b(X_i)$$ are functions of the same underlying random variables.

• Because I find your question baffling -- "what does it represent?" seems self-evident -- allow me to simplify the notation. Let $U$ stand for the random variable $A_i$ and $V$ stand for $B_j.$ You are asking "what does $E[UV]$ mean?" If that's not sufficiently clear, let $W$ stand for the random variable $UV$. Your question is "what does $E[W]$ mean"? Are you looking for a definition, some kind of interpretation, or something else??
– whuber
Dec 7, 2023 at 14:53
• @whuber You are right, the question is unclear. I've added details Dec 7, 2023 at 15:07
• Re the edits: again, simplifying the notation helps. Since $$E[W]=\int_{\Omega}W(\omega)\mathbb P(\mathrm d\omega),$$ substitution $W=A_iB_j$ and the definition $(A_iB_j)(\omega)=A_i(\omega)B_j(\omega)$ gives $$E[A_iB_j]=\int_{\Omega}A_i(\omega)B_j(\omega)\mathbb P(\mathrm d\omega).$$ For examples of non-independent random variables you need look no further than spaces $\Omega$ with two elements.
– whuber
Dec 7, 2023 at 15:26
• @whuber I'm confused, why is the sample space of $W$ simply $\Omega$ and not $\Omega\times\Omega$? Dec 7, 2023 at 15:36
• I don't see why your random variable $Z(\omega, \omega^\prime) = X(\omega)Y(\omega^\prime)$ is equivalent to a random variable $Z(\omega) = X(\omega)Y(\omega)$. These two random variables aren't even on the same probability space. And the way you have constructed the product space, it's bound to be that any two random variables $X$ and $Y$ will factor in expectation because the measure you have defined on the product space is a product measure! Product measures are the underlying measure theoretic basis for independence. Dec 7, 2023 at 15:39

Even in very advanced mathematics it helps to study simple examples. Part of the art of reading and learning mathematics is to construct such examples for yourself. This answer illustrates the process.

Let's consider the simplest non-trivial possible situation, where the probability space $$\Omega = \{\omega_1, \omega_2\}$$ has just two elements. We will suppose every subset of $$\Omega$$ is an event, which enables us to specify any probability distribution in terms of the probabilities $$\pi_i = \mathbb P(\{\omega_i\})$$ (with $$\pi_1+\pi_2=1,$$ of course).

To be concrete, let $$A:\Omega\to\mathbb R$$ be the random variable $$A(\omega_1) = 0$$ and $$A(\omega_2) = 4;$$ and let $$B$$ be the random variable $$B(\omega_1) = 2$$ and $$B(\omega_2) = 0.$$ (I chose these numbers to make the arithmetic simple, especially when working with the uniform distribution $$\pi_1=\pi_2=1/2.$$)

We can present this information about the random variables $$A$$ and $$B$$ conveniently as a table of probabilities and values of the random variables:

$$\begin{array}{cc|cc} \omega & \mathbb P & A & B\\ \hline \omega_1 &\pi_1 & 0 & 2\\ \omega_2 & \pi_2 & 4 & 0 \end{array}$$

To construct iid variables we need $$N$$ separate copies of this probability space. The simplest way, with $$N=2,$$ is to work in the product space $$\Omega\times \Omega.$$ The definitions tell us all the probabilities and the values of the random variables, so I will just summarize the results here, writing $$(\omega,\eta)$$ for a generic element of $$\Omega\times \Omega.$$ $$A_1$$ is the random variable defined by

$$A_1(\omega,\eta) = A(\omega)$$

while $$A_2$$ is the random variable

$$A_2(\omega,\eta) = A(\eta),$$

and likewise for the $$B_i.$$

$$\begin{array}{ccc|rrrr} \omega & \eta & \mathbb P & A_1 & A_2 & B_1 & B_2\\ \hline \omega_1 & \omega_1 & \pi_1^2 & 0 & 0 & 2 & 2\\ \omega_1 & \omega_2 & \pi_1\pi_2 & 0 & 4 & 2 & 0\\ \omega_2 & \omega_1 & \pi_2\pi_1 & 4 & 0 & 0 & 2\\ \omega_2 & \omega_2 & \pi_2^2 & 4 & 4 & 0 & 0 \end{array}$$

There's nothing new here: this is the standard product construction for creating iid variables. If you're unconvinced, verify that $$(A_1,A_2)$$ are independent and $$(B_1,B_2)$$ are independent.

The question concerns functions of these random variables. The left hand side of the equation in the question expands to

$$E\left[\frac{1}{N}\sum_{i=1}^N A_i\right] = E\left[\frac{1}{2}\left(A_1+A_2\right)\right].$$

It is convenient to denote this $$\bar A,$$ as in the question. $$\bar B$$ is similarly defined. Let's expand the preceding table to include these quantities as well as their product, $$\bar A \bar B = (\sum A_i \sum B_j)/N^2.$$ Looking ahead to the right hand side of the equation, I will also include the products $$A_iB_j:$$

$$\begin{array}{ccc|rrrr:cccccc:c} \omega & \eta & \mathbb P & A_1 & A_2 & B_1 & B_2 & A_1B_1 & A_1B_2 & A_2B_1 & A_2B_2 & \bar A & \bar B & \bar A \bar B\\ \hline \omega_1 & \omega_1 & \pi_1^2 & 0 & 0 & 2 & 2 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ \omega_1 & \omega_2 & \pi_1\pi_2 & 0 & 4 & 2 & 0 & 0 & 0 & 8 & 0 & 2 & 1 & 2\\ \omega_2 & \omega_1 & \pi_2\pi_1 & 4 & 0 & 0 & 2 & 0 & 8 & 0 & 0 & 2 & 1 & 2\\ \omega_2 & \omega_2 & \pi_2^2 & 4 & 4 & 0 & 0 & 0 & 0 & 0 & 0 & 4 & 0 & 0 \end{array}$$

There is nothing probabilistic about these calculations. They are carried out separately on each row and are completely deterministic.

The integrals are just sums of values times their probabilities, as always:

\begin{aligned} E[\bar A\bar B] &= \iint_{\Omega\times \Omega} \bar A(\omega,\eta)\bar B(\omega,\eta)\mathbb (\mathbb P\otimes\mathbb P)(\mathrm d(\omega,\eta))\\ & = \pi_1^2(0) + \pi_1\pi_2(2) + \pi_2\pi_1(2) + \pi_2^2(0) = 2\pi_1\pi_2. \end{aligned}\tag{*}

That's the left hand side of the equation in the question. The right hand side is a combination of four expectations. They are worked out in the same way from the table, by multiplying suitable columns by their probabilities and summing:

\begin{aligned} E[A_1B_1] &= \pi_1^2(0) + \pi_2\pi_1(0) + \pi_1\pi_2(0) + \pi_2^2(0) &= 0\\ E[A_1B_2] &= \cdots &= \pi_2\pi_1(8)\\ E[A_2B_1] &= \cdots &= \pi_1\pi_2(8)\\ E[A_2B_2] &= \cdots &= 0. \end{aligned}

Consequently -- still just doing simple algebra,

$$\frac{1}{N^2}\sum_i\sum_j E[A_iB_j] = \frac{1}{4}\left(0 + \pi_2\pi_1(8) + \pi_1\pi_2(8) + 0)\right) = 2\pi_1\pi_2,$$

agreeing with $$(*)$$ and demonstrating the equation in this case.

Finally, compute $$E[\bar A] = 4\pi_2^2 + 4\pi_1\pi_2$$ and $$E[\bar B] = 2\pi_1\pi_2.$$ For most values of $$\pi_1$$ the product of these expectations does not equal $$E[\bar A\bar B] = 2\pi_1\pi_2:$$ that is because $$A$$ and $$B$$ are not independent. (Indeed, $$B = (4-A)/2$$ exhibits this dependence explicitly.)

• This is really great, thank you so much for putting in the effort of doing this. I really appreciate it. Dec 7, 2023 at 18:58