# Variance of sum of random number of random variables (Cambridge University Worksheet)

In the vein of my last question, I'm now at a roadblock on question 3 of this sheet:

http://www.trin.cam.ac.uk/dpk10/IA/exsheet3.pdf

(note: it's not my intention to ask every question I get stuck on here, merely the ones which have interesting general results; it just so happens that the two I've struggled with so far fit this criteria)

It goes as follows:

Let $N$ be a non-negative integer-valued random variable with mean $\mu_1$ and variance $\sigma_1^2$, and let $X_1, X_2, ...$ be random variables, each with mean $\mu_2$ and variance $\sigma_2^2$; furthermore, assume that $N, X_1, X_2, . . .$ are independent. Without using generating functions, calculate the mean and variance of the random variable $S_N = X_1 + ... + X_N$ (when $N=0$ interpret $S_N$ as $0$).

I have the answer to calculating the mean, which I'll write up as an answer for future reference. Here's where I've got to with the variance bit:

$\mathbb{E}(S_N^2) = 0.P(N=0) + \sum_{r=1}^\infty\mathbb{E}((\sum_{i=1}^rX_i)^2).P(N=r)$

For $r=1$, we have the inner expectation as: $\mathbb{E}(X_1^2)$ which is equal to $\sigma_2^2+\mu_2^2$.

For $r > 1$, we have the inner expectation equal to: $r(\sigma_2^2 + 2\mu_2^2)$

So $\mathbb{E}(S_N^2) = (\sigma_2^2+\mu_2^2)P(N=1) + (\sigma_2^2 + 2\mu_2^2)\sum_{r=2}^\infty rP(N=r)$

which is almost in the form where I can use the same trick as in calculating the mean, but not quite.

Any help is much appreciated.

• Have you tried using the law of total variance? Commented Nov 2, 2013 at 20:45
• This looks like a really sharp way to deal with the question, but I think I'm applying conditional variance/expectation incorrectly. Could you please walk me through it? Commented Nov 3, 2013 at 0:06
• I think you have an error in your $E(S_N^2)$ there. Commented Nov 3, 2013 at 1:56
• You still haven't got it by the look. Check my updated answer. Commented Nov 3, 2013 at 2:19
• Ah, I see. I was still counting the number of cross terms incorrectly. Of course there are $r^2$ total terms of which $r$ are of the form $X_i^2$ and the remaining $r^2-r$ are of the form $X_iX_j$. Thank you very much for helping me figure it out at last. Commented Nov 3, 2013 at 12:50

The easy way is to use the law of total variance:

$$\text{Var}(S) = E_N\left[\text{Var}(S|N)\right] + \text{Var}_N\left[E(S|N)\right] =\text{E}_N\left[N\cdot \text{Var}(X)\right] + \text{Var}_N\left[N\cdot\text{E}(X)\right]$$

Can you do it from there? It's pretty much just substitution (well, that and really basic properties of expectation and variance).

(The first part is even more straightforward using the law of total expectation.)

--

As Spy_Lord notes, the answer is $\text{E}(N)\cdot \text{Var}(X) + \text{Var}(N)\cdot\text{E}(X)^2$

Alternative approach is to evaluate $E(S_N^2)$. Following the approach you seem to be aiming at:

\begin{eqnarray} E(S_N^2) &=& \sum_r E(S_N^2|N=r) p_r\\ &=& \sum_r (r\sigma_2^2+r^2 \mu_2^2) p_r\\ &=& \sigma_2^2\sum_r rp_r+\mu_2^2\sum_rr^2 p_r \\ &=& \sigma_2^2 \text{E}N+\mu_2^2\text{E}(N^2) \end{eqnarray}

and I assume you can do it from there.

However, to be honest, I think this way is easier (it's actually the same approach, you just don't need to sum over all the mutually exclusive events that way). The law of total expectation says $\text{E}(X) = \text{E}_Y[\text{E}_{X|Y}(X|Y)]$, so

\begin{eqnarray} \text{E}(S^2_N) &=& \text{E}_N[\text{E}(S^2_N|N)]\\ &=& \text{E}_N[N\sigma_2^2+N^2\mu_2^2]\\ &=& \sigma_2^2\text{E}(N)+\mu_2^2\text{E}(N^2) \end{eqnarray}

• Yes, thank you for the help - I was forgetting that the N inside the brackets was still a random variable. So the answer is $Var(S_N) = E_N(N.\sigma_2^2) + Var_N(N.\mu_2) = \sigma_2^2E_N(N) + \mu_2^2Var_N(N) = \sigma_2^2\mu_1 + \mu_2^2\sigma_1^2$ Commented Nov 3, 2013 at 1:20
• It is indeed very slick using total expectation/variance to deal with this question. This is not remotely necessary, but do you think there's a way to make my initial approach work? Commented Nov 3, 2013 at 1:22
• Yes, you have the answer in that comment right. I'll edit that back in to my answer now I know I'm not just giving you the answer. I'll have a look at what you're doing. I expect it can be made to work rather easily. Commented Nov 3, 2013 at 1:26

The law of total variance is the easiest way to do this. But there are several occasions when we don't know how many random variables we are dealing with (e.g. branching processes such as Galton-Watson, birth-death processes, queues) where probability-generating functions are a useful technique. It is possible to derive mean and variance using the PGF, so I want to demonstrate how this can serve as an alternative. Why bother? One motivation is that this method will generalize easily to find any factorial moment, and hence any moment, of distribution.

A few general results: a PGF $G_X(z)=\mathbb{E}(z^X)$ has $\lim_{z\uparrow 1}\, G_X(z)=\lim_{z\uparrow 1}\, \mathbb{E}(z^X)=1$. Factorial moments are found by taking the limit of the appropriate derivative of the PGF as $z$ goes to 1 from below. So for a random variable $X$:

\begin{eqnarray} \mathbb{E}(X) &=& \lim_{z\uparrow 1}\, G'_X(z)\\ \mathbb{E}(X(X-1)) &=& \lim_{z\uparrow 1}\, G''_X(z)\\ \mathbb{E}(X(X-1)(X-2)) &=& \lim_{z\uparrow 1}\, G'''_X(z) \end{eqnarray}

And so on for higher moments. The key here is that if $S_N=\sum_{i=1}^N X_i$ with iid $X_i$ then $G_{S_N}(z)=G_N(G_X(z))$. Proof:

\begin{eqnarray} G_{S_N}(z) &=& \mathbb{E}_N(\mathbb{E}(z^{\sum_{i=1}^N X_i})) &=& \mathbb{E}_N(\mathbb{E}(\prod_{i=1}^N z^{X_i})) &=& \mathbb{E}_N(\prod_{i=1}^N (\mathbb{E}(z^{X_i})) \\ &=& \mathbb{E}_N(\prod_{i=1}^N G_X(z)) &=& \mathbb{E}_N(G_X(z)^N) &=& G_N(G_X(z)) \end{eqnarray}

Also note $\lim_{z\uparrow 1}\, G_X(z)=\lim_{z\uparrow 1}\, G_N(z)=1$, $\lim_{z\uparrow 1}\, G'_X(z)=\mu_X$, $\lim_{z\uparrow 1}\, G'_N(z)=\mu_N$, $\lim_{z\uparrow 1}\, G''_X(z)=\mathbb{E}(X^2-X)=\sigma_X^2+\mu_X^2-\mu_X$ and $\lim_{z\uparrow 1}\, G''_N(z)=\sigma_N^2+\mu_N^2-\mu_N$.

Since $G_{S_N}(z)=G_N(G_X(z))$ we can use the chain rule to find the mean and variance of $S_N$:

\begin{eqnarray} \mathbb{E}(S_N) &=& \lim_{z\uparrow 1}\, \frac{d}{dz}G_N(G_X(z))=\lim_{z\uparrow 1}\, G'_X(z)G'_N(G_X(z))=\mu_X \mu_N\\ \mathbb{E}(S_N(S_N-1)) &=& \lim_{z\uparrow 1}\, \frac{d^2}{dz^2}G_N(G_X(z))\\ \mathbb{E}(S_N^2-S_N) &=& \lim_{z\uparrow 1}\, \left(G''_X(z)G'_N(G_X(z))+G'_X(z)^2 G''_N(G_X(z))\right) \\ \mathbb{E}(S_N^2)-\mu_X \mu_N &=& (\sigma_X^2+\mu_X^2-\mu_X)(\mu_N)+(\mu_X)^2(\sigma_N^2+\mu_N^2-\mu_N) \\ \mathbb{E}(S_N^2) &=& \mu_N \sigma_X^2 + \mu_X^2 \sigma_N^2 + \mu_X^2 \mu_N^2 \\ \operatorname{Var}(S_N) &=& \mathbb{E}(S_N^2)-\mathbb{E}(S_N)^2=\mu_N \sigma_X^2 + \mu_X^2 \sigma_N^2 + \mu_X^2 \mu_N^2-(\mu_X \mu_N)^2 \\ \operatorname{Var}(S_N) &=& \mu_N \sigma_X^2 + \mu_X^2 \sigma_N^2 \end{eqnarray}

It's a little gruesome and there's no doubt the law of total variance is easier. But if the standard results are taken for granted, this is only a couple of lines of algebra and calculus, and I've given more detail than some of the other answers which makes it look worse than it is. If you wanted the higher moments, this is a viable approach.

The solution to calculating the mean:

$\mathbb{E}(S_N) = 0.P(N=0) + \mathbb{E}(X_1).P(N=1) + \mathbb{E}(X_1+X_2).P(N=2) + . . .$

$= 0 + \mu_2P(N=1) + 2\mu_2P(N=2) + . . . = \mu_2\sum_{i=0}^\infty i.P(N=i)$

and the infinite sum above is just equal to the expectation of $N$, hence:

$\mathbb{E}(S_N) = \mu_2.\mathbb{E}(N) = \mu_2\mu_1$

• To find $\mathbb{E}(S_N^2),$ why not apply this answer to the variables $X_1^2, X_2^2, \ldots, X_N^2, \ldots$? You know their means and they are obviously independent of each other and of $N$, so aren't you done?
– whuber
Commented Nov 2, 2013 at 21:05
• I'm not sure I follow. $\mathbb{E}(S_N^2)$ is not just $= X_1^2 + X_2^2 + ... + X_N^2$, and not even equal to that with terms like $2X_1X_2 + 2X_1X_3 + . . .$. I believe it is equal to the first equality in my question (I could very well be mistaken of course). [I do actually apply this answer to calculate $\mathbb{E}(X_1^2)$ and so on in my $r=1, r>1$ cases above] Commented Nov 3, 2013 at 0:05