8
$\begingroup$

In another installment of intuitions for identities in probability, consider the elementary identity Law of Total Variance

$$ \begin{eqnarray} \rm{Var}(X) &=&\rm{E}[\rm{Var}(X|Y)] + \rm{Var}(E[X|Y]) \end{eqnarray} $$

It is a simple straightforward algebraic manipulation of the definition of moments into summation, or, as in the wikipedia link, via manipulation of E and Var.

But this identity, I have no idea what it means. I suppose it means you could presumably calculate the variance of one variable using another variable to help out, but it doesn't look like it simplifies things or makes things more tractable.

The wiki page says

first component is called the expected value of the process variance (EVPV) and the second is called the variance of the hypothetical means (VHM)

which is as enlightening as reading off names can be.

So what does it really mean? Is there an intuition about the two parts? Do you need an intuition of $E[E[X|Y]] = E[X]$ first? A geometric intuition might be nice, but also a wordy explanation, little algebra, would help immensely.

Are there any good linear algebra interpretations or physical interpretations or other that would give insight into this identity?

Improve this question
$\endgroup$
2
  • 4
    $\begingroup$ but it doesn't look like it simplifies things or makes things more tractable – it's useful in basically any case where there's some auxiliary $Y$ that makes $X \mid Y$ easier to think about than just $X$ itself. For example, take$\DeclareMathOperator{\E}{\mathbb E}\DeclareMathOperator{\Var}{Var}$ $X \mid Y \sim \mathcal{N}(Y, Y \sigma^2)$ and $Y \sim \mathrm{Binomial}(n, p)$; then \begin{gather}\E[\Var(X \mid Y)] = \E[Y \sigma^2] = n p \sigma^2 \\ \Var(\E[X \mid Y]) = \Var(Y) = n p (1-p).\end{gather} Trying to calculate that directly would have been much less straightforward. $\endgroup$ – Danica Jul 13 '18 at 16:45
  • $\begingroup$ related: stats.stackexchange.com/questions/71620/… $\endgroup$ – Taylor Jul 13 '18 at 21:34
5
$\begingroup$

To get some simple intuition, we will compare with a two-way analysis of variance. Let $Y_{ij} = \mu_i +\epsilon_{ij}$ where the $\epsilon_{ij}$ are iid with expectation zero and common variance $\sigma^2$, $i=1,\dotsc,k; j=1,\dotsc,n_i$.

Then we have the decomposition $$ \sum_{i=1}^k\sum_{j=1}^{n_i} (Y_{ij}-\overline{\overline{Y}})^2 = \sum_{i=1}^k\sum_{j=1}^{n_i} (Y_{ij}-\overline{Y_i})^2 + \sum_{i=1}^k n_i((\overline{Y_{i}}-\overline{\overline{Y}})^2 $$ where the first term on the right measures within-group variance (and can be used to estimate the common within-group variance $\sigma^2$), the second term measures between-group variance, and can be used to estimate $\sigma^2$ only under the hypothesis that all the $\mu_i$ have a common value. Otherwise, it will contain an extra component, the "variance of the $\mu_i$'s". This has the same form as the law of total variance!

Formally, let group membership be the random variable $G$. Then we get $$ Var Y = E Var (Y | G) + Var E (Y | G) $$ and we can read this as "variance of $Y$ is the expected value of within-group variance plus the variance of group expectations." That is the same as our interpretation of the ANOVA decomposition above. Looking closer at the derivation (which we didn't give here) you can see that it is really a version of the Pythagorean theorem. For that point of view see Law of total variance as Pythagorean theorem

Improve this answer
$\endgroup$
2
  • $\begingroup$ My mathjax hack to get double overlines do not work very well. Ideas to get it better? $\endgroup$ – kjetil b halvorsen♦ Jul 13 '18 at 22:14
  • 1
    $\begingroup$ Double overline looks much better with $X$ than with $Y$ for some reason: $\overline{\overline{X}}$. $\endgroup$ – jbowman Jul 14 '18 at 0:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.