Intuition (geometric or other) of $Var(X) = E[X^2] - (E[X])^2$

Question

Consider the elementary identity of variance:

$$ \begin{eqnarray} Var(X) &=& E[(X - E[X])^2]\\ &=& ...\\ &=& E[X^2] - (E[X])^2 \end{eqnarray} $$

It is a simple algebraic manipulation of the definition of a central moment into non-central moments.

It allows convenient manipulation of $Var(X)$ in other contexts. It also allows calculation of variance via a single pass over data rather than two passes, first to calculate the mean, and then to calculate the variance.

But what does it mean? To me there's no immediate geometric intuition that relates spread about the mean to spread about 0. As $X$ is a set on a single dimension, how do you view the spread around a mean as the difference between spread around the origin and the square of the mean?

Are there any good linear algebra interpretations or physical interpretations or other that would give insight into this identity?

Answer 1

Expanding on @whuber's point in the comments, if $Y$ and $Z$ are orthogonal, you have the Pythagorean Theorem:

$$ \|Y\|^2 + \|Z\|^2 = \|Y + Z\|^2 $$

Observe that $\langle Y, Z \rangle \equiv \mathrm{E}[YZ]$ is a valid inner product and that $\|Y\| = \sqrt{\mathrm{E}[Y^2]}$ is the norm induced by that inner product.

Let $X$ be some random variable. Let $Y = \mathrm{E}[X]$, Let $Z = X - \mathrm{E}[X]$. If $Y$ and $Z$ are orthogonal:

\begin{align*} & \|Y\|^2 + \|Z\|^2 = \|Y + Z\|^2 \\ \Leftrightarrow \quad&\mathrm{E}[\mathrm{E}[X]^2] + \mathrm{E}[(X - \mathrm{E}[X])^2] = \mathrm{E}[X^2] \\ \Leftrightarrow \quad & \mathrm{E[X]}^2 + \mathrm{Var}[X]= \mathrm{E}[X^2] \end{align*}

And it's easy to show that $Y = \mathrm{E}[X]$ and $Z = X - \mathrm{E}[X]$ are orthogonal under this inner product:

$$\langle Y, Z \rangle = \mathrm{E}[\mathrm{E}[X]\left(X - \mathrm{E}[X] \right)] = \mathrm{E}[X]^2 - \mathrm{E}[X]^2 = 0$$

One of the legs of the triangle is $X - \mathrm{E}[X]$, the other leg is $\mathrm{E}[X]$, and the hypotenuse is $X$. And the Pythagorean theorem can be applied because a demeaned random variable is orthogonal to its mean.

---

Technical remark:

$Y$ in this example really should be the vector $Y = \mathrm{E}[X] \mathbf{1}$, that is, the scalar $\mathrm{E}[X]$ times the constant vector $\mathbf{1}$ (e.g. $\mathbf{1} = [1, 1, 1, \ldots, 1]'$ in the discrete, finite outcome case). $Y$ is the vector projection of $X$ onto the constant vector $\mathbf{1}$.

Simple Example

Consider the case where $X$ is a Bernoulli random variable where $p = .2$. We have:

$$ X = \begin{bmatrix} 1 \\ 0 \end{bmatrix} \quad P = \begin{bmatrix} .2 \\ .8 \end{bmatrix} \quad \mathrm{E}[X] = \sum_i P_iX_i = .2 $$

$$ Y = \mathrm{E}[X]\mathbf{1} = \begin{bmatrix} .2 \\ .2 \end{bmatrix} \quad Z = X - \mathrm{E}[X] = \begin{bmatrix} .8 \\ -.2 \end{bmatrix} $$

And the picture is:

The squared magnitude of the red vector is the variance of $X$, the squared magnitude of the blue vector is $\mathrm{E}[X]^2$, and the squared magnitude of the yellow vector is $\mathrm{E}[X^2]$.

REMEMBER though that these magnitudes, the orthogonality etc... aren't with respect to the usual dot product $\sum_i Y_iZ_i$ but the inner product $\sum_i P_iY_iZ_i$. The magnitude of the yellow vector isn't 1, it is .2.

The red vector $Y = \mathrm{E}[X]$ and the blue vector $Z = X - \mathrm{E}[X]$ are perpendicular under the inner product $\sum_i P_i Y_i Z_i$ but they aren't perpendicular in the intro, high school geometry sense. Remember we're not using the usual dot product $\sum_i Y_i Z_i$ as the inner product!

Answer 2

I will go for a purely geometric approach for a very specific scenario. Let us consider a discrete valued random variable $X$ taking values $\{x_1,x_2\}$ with probabilities $(p_1,p_2)$. We will further assume that this random variable can be represented in $\mathbb{R}^2$ as a vector, $\mathbf{X} = \left(x_1\sqrt{p_1},x_2\sqrt{p_2} \right)$.

Notice that the length-square of $\mathbf{X}$ is $x_1^2p_1+x_2^2p_2$ which is equal to $E[X^2]$. Thus, $\left\| \mathbf{X} \right\| = \sqrt{E[X^2]}$.

Since $p_1+p_2=1$, the tip of vector $\mathbf{X}$ actually traces an ellipse. This becomes easier to see if one reparametrizes $p_1$ and $p_2$ as $\cos^2(\theta)$ and $\sin^2(\theta)$. Hence, we have $\sqrt{p_1} =\cos(\theta)$ and $\sqrt{p_2} = \sin(\theta)$.

One way of drawing ellipses is via a mechanism called Trammel of Archimedes. As described in wiki: It consists of two shuttles which are confined ("trammelled") to perpendicular channels or rails, and a rod which is attached to the shuttles by pivots at fixed positions along the rod. As the shuttles move back and forth, each along its channel, the end of the rod moves in an elliptical path. This principle is illustrated in the figure below.

Now let us geometrically analyze one instance of this trammel when the vertical shuttle is at $A$ and the horizontal shuttle is at $B$ forming an angle of $\theta$. Due to construction, $\left|BX\right| = x_2$ and $\left| AB \right| = x_1-x_2$, $\forall \theta$ (here $x_1\geq x_2$ is assumed wlog).

Let us draw a line from origin, $OC$, that is perpendicular to the rod. One can show that $\left| OC \right|=(x_1-x_2) \sin(\theta) \cos(\theta)$. For this specific random variable \begin{eqnarray} Var(X) &=& (x_1^2p_1 +x_2^2p_2) - (x_1p_1+x_2p_2)^2 \\ &=& x_1^2p_1 +x_2^2p_2 - x_1^2p_1^2 - x_2^2p_2^2 - 2x_1x_2p_1p_2 \\ &=& x_1^2(p_1-p_1^2) + x_2^2(p_2-p_2^2) - 2x_1x_2p_1p_2 \\ &=& p_1p_2(x_1^2- 2x_1x_2 + x_2^2) \\ &=& \left[(x_1-x_2)\sqrt{p_1}\sqrt{p_2}\right]^2 = \left|OC \right|^2 \end{eqnarray} Therefore, the perpendicular distance $\left|OC \right|$ from the origin to the rod is actually equal to the standard deviation, $\sigma$.

If we compute the length of segment from $C$ to $X$: \begin{eqnarray} \left|CX\right| &=& x_2 + (x_1-x_2)\cos^2(\theta) \\ &=& x_1\cos^2(\theta) +x_2\sin^2(\theta) \\ &=& x_1p_1 + x_2p_2 = E[X] \end{eqnarray}

Applying the Pythagorean Theorem in the triangle OCX, we end up with \begin{equation} E[X^2] = Var(X) + E[X]^2. \end{equation}

To summarize, for a trammel that describes all possible discrete valued random variables taking values $\{x_1,x_2\}$, $\sqrt{E[X^2]}$ is the distance from the origin to the tip of the mechanism and the standard deviation $\sigma$ is the perpendicular distance to the rod.

Note: Notice that when $\theta$ is $0$ or $\pi/2$, $X$ is completely deterministic. When $\theta$ is $\pi/4$ we end up with maximum variance.

Answer 3

You can rearrange as follows:

$$ \begin{eqnarray} Var(X) &=& E[X^2] - (E[X])^2\\ E[X^2] &=& (E[X])^2 + Var(X) \end{eqnarray} $$

Then, interpret as follows: the expected square of a random variable is equal to the square of its mean plus the expected squared deviation from its mean.

Answer 4

The general intuition is that you can relate these moments using the Pythagorean Theorem (PT) in a suitably defined vector space, by showing that two of the moments are perpendicular and the third is the hypotenuse. The only algebra needed is to show that the two legs are indeed orthogonal.

For the sake of the following I'll assume you meant sample means and variances for computation purposes rather than moments for full distributions. That is:

$$ \begin{array}{rcll} E[X] &=& \frac{1}{n}\sum x_i,& \rm{mean, first\ central\ sample\ moment}\\ E[X^2] &=& \frac{1}{n}\sum x^2_i,& \rm{second\ sample\ moment\ (non-central)}\\ Var(X) &=& \frac{1}{n}\sum (x_i - E[X])^2,& \rm{variance, second\ central\ sample\ moment} \end{array} $$

(where all sums are over $n$ items).

For reference, the elementary proof of $Var(X) = E[X^2] - E[X]^2$ is just symbol pushing: $$ \begin{eqnarray} Var(X) &=& \frac{1}{n}\sum (x_i - E[X])^2\\ &=& \frac{1}{n}\sum (x^2_i - 2 E[X]x_i + E[X]^2)\\ &=& \frac{1}{n}\sum x^2_i - \frac{2}{n} E[X] \sum x_i + \frac{1}{n}\sum E[X]^2\\ &=& E[X^2] - 2 E[X]^2 + \frac{1}{n} n E[X]^2\\ &=& E[X^2] - E[X]^2\\ \end{eqnarray} $$

There's little meaning here, just elementary manipulation of algebra. One might notice that $E[X]$ is a constant inside the summation, but that is about it.

Now in the vector space/geometrical interpretation/intuition, what we'll show is the slightly rearranged equation that corresponds to PT, that

$$ \begin{eqnarray} Var(X) + E[X]^2 &=& E[X^2] \end{eqnarray} $$

So consider $X$, the sample of $n$ items, as a vector in $\mathbb{R}^n$. And let's create two vectors $E[X]{\bf 1}$ and $X-E[X]{\bf 1}$.

The vector $E[X]{\bf 1}$ has the mean of the sample as every one of its coordinates.

The vector $X-E[X]{\bf 1}$ is $\langle x_1-E[X], \dots, x_n-E[X]\rangle$.

These two vectors are perpendicular because the dot product of the two vectors turns out to be 0: $$ \begin{eqnarray} E[X]{\bf 1}\cdot(X-E[X]{\bf 1}) &=& \sum E[X](x_i-E[X])\\ &=& \sum (E[X]x_i-E[X]^2)\\ &=& E[X]\sum x_i - \sum E[X]^2\\ &=& n E[X]E[X] - n E[X]^2\\ &=& 0\\ \end{eqnarray} $$

So the two vectors are perpendicular which means they are the two legs of a right triangle.

Then by PT (which holds in $\mathbb{R}^n$), the sum of the squares of the lengths of the two legs equals the square of the hypotenuse.

By the same algebra used in the boring algebraic proof at the top, we showed that we get that $E[X^2]$ is the square of the hypotenuse vector:

$(X-E[X])^2 + E[X]^2 = ... = E[X^2]$ where squaring is the dot product (and it's really $E[x]{\bf 1}$ and $(X-E[X])^2$ is $Var(X)$.

The interesting part about this interpretation is the conversion from a sample of $n$ items from a univariate distribution to a vector space of $n$ dimensions. This is similar to $n$ bivariate samples being interpreted as really two samples in $n$ variables.

In one sense that is enough, the right triangle from vectors and $E[X^2]$ pops out as the hypotnenuse. We gave an interpretation (vectors) for these values and show they correspond. That's cool enough, but unenlightening either statistically or geometrically. It wouldn't really say why and would be a lot of extra conceptual machinery to, in the end mostly, reproduce the purely algebraic proof we already had at the beginning.

Another interesting part is that the mean and variance, though they intuitively measure center and spread in one dimension, are orthogonal in $n$ dimensions. What does that mean, that they're orthogonal? I don't know! Are there other moments that are orthogonal? Is there a larger system of relations that includes this orthogonality? central moments vs non-central moments? I don't know!

Answer 5

Sorry for not having the skill to elaborate and provide a proper answer, but I think the answer lies in the physical classical mechanics concept of moments, especially the conversion between 0 centred "raw" moments and mean centred central moments. Bear in mind that variance is the second order central moment of a random variable.