Variance of sum of random number of random variables (Cambridge University Worksheet)

Question

In the vein of my last question, I'm now at a roadblock on question 3 of this sheet:

http://www.trin.cam.ac.uk/dpk10/IA/exsheet3.pdf

(note: it's not my intention to ask every question I get stuck on here, merely the ones which have interesting general results; it just so happens that the two I've struggled with so far fit this criteria)

It goes as follows:

Let $N$ be a non-negative integer-valued random variable with mean $\mu_1$ and variance $\sigma_1^2$, and let $X_1, X_2, ...$ be random variables, each with mean $\mu_2$ and variance $\sigma_2^2$; furthermore, assume that $N, X_1, X_2, . . .$ are independent. Without using generating functions, calculate the mean and variance of the random variable $S_N = X_1 + ... + X_N$ (when $N=0$ interpret $S_N$ as $0$).

I have the answer to calculating the mean, which I'll write up as an answer for future reference. Here's where I've got to with the variance bit:

$\mathbb{E}(S_N^2) = 0.P(N=0) + \sum_{r=1}^\infty\mathbb{E}((\sum_{i=1}^rX_i)^2).P(N=r)$

For $r=1$, we have the inner expectation as: $\mathbb{E}(X_1^2)$ which is equal to $\sigma_2^2+\mu_2^2$.

For $r > 1$, we have the inner expectation equal to: $r(\sigma_2^2 + 2\mu_2^2)$

So $\mathbb{E}(S_N^2) = (\sigma_2^2+\mu_2^2)P(N=1) + (\sigma_2^2 + 2\mu_2^2)\sum_{r=2}^\infty rP(N=r)$

which is almost in the form where I can use the same trick as in calculating the mean, but not quite.

Any help is much appreciated.

Answer 1

The easy way is to use the law of total variance:

$$\text{Var}(S) = E_N\left[\text{Var}(S|N)\right] + \text{Var}_N\left[E(S|N)\right] =\text{E}_N\left[N\cdot \text{Var}(X)\right] + \text{Var}_N\left[N\cdot\text{E}(X)\right]$$

Can you do it from there? It's pretty much just substitution (well, that and really basic properties of expectation and variance).

(The first part is even more straightforward using the law of total expectation.)

--

As Spy_Lord notes, the answer is $\text{E}(N)\cdot \text{Var}(X) + \text{Var}(N)\cdot\text{E}(X)^2$

---

Alternative approach is to evaluate $E(S_N^2)$. Following the approach you seem to be aiming at:

\begin{eqnarray} E(S_N^2) &=& \sum_r E(S_N^2|N=r) p_r\\ &=& \sum_r (r\sigma_2^2+r^2 \mu_2^2) p_r\\ &=& \sigma_2^2\sum_r rp_r+\mu_2^2\sum_rr^2 p_r \\ &=& \sigma_2^2 \text{E}N+\mu_2^2\text{E}(N^2) \end{eqnarray}

and I assume you can do it from there.

However, to be honest, I think this way is easier (it's actually the same approach, you just don't need to sum over all the mutually exclusive events that way). The law of total expectation says $\text{E}(X) = \text{E}_Y[\text{E}_{X|Y}(X|Y)]$, so

\begin{eqnarray} \text{E}(S^2_N) &=& \text{E}_N[\text{E}(S^2_N|N)]\\ &=& \text{E}_N[N\sigma_2^2+N^2\mu_2^2]\\ &=& \sigma_2^2\text{E}(N)+\mu_2^2\text{E}(N^2) \end{eqnarray}

Answer 2

The law of total variance is the easiest way to do this. But there are several occasions when we don't know how many random variables we are dealing with (e.g. branching processes such as Galton-Watson, birth-death processes, queues) where probability-generating functions are a useful technique. It is possible to derive mean and variance using the PGF, so I want to demonstrate how this can serve as an alternative. Why bother? One motivation is that this method will generalize easily to find any factorial moment, and hence any moment, of distribution.

A few general results: a PGF $G_X(z)=\mathbb{E}(z^X)$ has $\lim_{z\uparrow 1}\, G_X(z)=\lim_{z\uparrow 1}\, \mathbb{E}(z^X)=1$. Factorial moments are found by taking the limit of the appropriate derivative of the PGF as $z$ goes to 1 from below. So for a random variable $X$:

\begin{eqnarray} \mathbb{E}(X) &=& \lim_{z\uparrow 1}\, G'_X(z)\\ \mathbb{E}(X(X-1)) &=& \lim_{z\uparrow 1}\, G''_X(z)\\ \mathbb{E}(X(X-1)(X-2)) &=& \lim_{z\uparrow 1}\, G'''_X(z) \end{eqnarray}

And so on for higher moments. The key here is that if $S_N=\sum_{i=1}^N X_i$ with iid $X_i$ then $G_{S_N}(z)=G_N(G_X(z))$. Proof:

\begin{eqnarray} G_{S_N}(z) &=& \mathbb{E}_N(\mathbb{E}(z^{\sum_{i=1}^N X_i})) &=& \mathbb{E}_N(\mathbb{E}(\prod_{i=1}^N z^{X_i})) &=& \mathbb{E}_N(\prod_{i=1}^N (\mathbb{E}(z^{X_i})) \\ &=& \mathbb{E}_N(\prod_{i=1}^N G_X(z)) &=& \mathbb{E}_N(G_X(z)^N) &=& G_N(G_X(z)) \end{eqnarray}

Also note $\lim_{z\uparrow 1}\, G_X(z)=\lim_{z\uparrow 1}\, G_N(z)=1$, $\lim_{z\uparrow 1}\, G'_X(z)=\mu_X$, $\lim_{z\uparrow 1}\, G'_N(z)=\mu_N$, $\lim_{z\uparrow 1}\, G''_X(z)=\mathbb{E}(X^2-X)=\sigma_X^2+\mu_X^2-\mu_X$ and $\lim_{z\uparrow 1}\, G''_N(z)=\sigma_N^2+\mu_N^2-\mu_N$.

Since $G_{S_N}(z)=G_N(G_X(z))$ we can use the chain rule to find the mean and variance of $S_N$:

\begin{eqnarray} \mathbb{E}(S_N) &=& \lim_{z\uparrow 1}\, \frac{d}{dz}G_N(G_X(z))=\lim_{z\uparrow 1}\, G'_X(z)G'_N(G_X(z))=\mu_X \mu_N\\ \mathbb{E}(S_N(S_N-1)) &=& \lim_{z\uparrow 1}\, \frac{d^2}{dz^2}G_N(G_X(z))\\ \mathbb{E}(S_N^2-S_N) &=& \lim_{z\uparrow 1}\, \left(G''_X(z)G'_N(G_X(z))+G'_X(z)^2 G''_N(G_X(z))\right) \\ \mathbb{E}(S_N^2)-\mu_X \mu_N &=& (\sigma_X^2+\mu_X^2-\mu_X)(\mu_N)+(\mu_X)^2(\sigma_N^2+\mu_N^2-\mu_N) \\ \mathbb{E}(S_N^2) &=& \mu_N \sigma_X^2 + \mu_X^2 \sigma_N^2 + \mu_X^2 \mu_N^2 \\ \operatorname{Var}(S_N) &=& \mathbb{E}(S_N^2)-\mathbb{E}(S_N)^2=\mu_N \sigma_X^2 + \mu_X^2 \sigma_N^2 + \mu_X^2 \mu_N^2-(\mu_X \mu_N)^2 \\ \operatorname{Var}(S_N) &=& \mu_N \sigma_X^2 + \mu_X^2 \sigma_N^2 \end{eqnarray}

It's a little gruesome and there's no doubt the law of total variance is easier. But if the standard results are taken for granted, this is only a couple of lines of algebra and calculus, and I've given more detail than some of the other answers which makes it look worse than it is. If you wanted the higher moments, this is a viable approach.

Answer 3

The solution to calculating the mean:

$\mathbb{E}(S_N) = 0.P(N=0) + \mathbb{E}(X_1).P(N=1) + \mathbb{E}(X_1+X_2).P(N=2) + . . .$

$= 0 + \mu_2P(N=1) + 2\mu_2P(N=2) + . . . = \mu_2\sum_{i=0}^\infty i.P(N=i)$

and the infinite sum above is just equal to the expectation of $N$, hence:

$\mathbb{E}(S_N) = \mu_2.\mathbb{E}(N) = \mu_2\mu_1$