18
$\begingroup$

Consider the elementary identity of variance:

$$ \begin{eqnarray} Var(X) &=& E[(X - E[X])^2]\\ &=& ...\\ &=& E[X^2] - (E[X])^2 \end{eqnarray} $$

It is a simple algebraic manipulation of the definition of a central moment into non-central moments.

It allows convenient manipulation of $Var(X)$ in other contexts. It also allows calculation of variance via a single pass over data rather than two passes, first to calculate the mean, and then to calculate the variance.

But what does it mean? To me there's no immediate geometric intuition that relates spread about the mean to spread about 0. As $X$ is a set on a single dimension, how do you view the spread around a mean as the difference between spread around the origin and the square of the mean?

Are there any good linear algebra interpretations or physical interpretations or other that would give insight into this identity?

$\endgroup$
  • 7
    $\begingroup$ Hint: this is the Pythagorean Theorem. $\endgroup$ – whuber Jan 3 '17 at 18:48
  • 1
    $\begingroup$ @Matthew I wonder what "$E$" is intended to mean. I suspect it is not an expectation, but just shorthand for the arithmetic mean. Otherwise the equations would be incorrect (and nearly meaningless, since they would then equate random variables with numbers). $\endgroup$ – whuber Jan 3 '17 at 19:00
  • 2
    $\begingroup$ @whuber Since inner products introduce the idea of distances and angles, and the inner product of the vector space of real-valued random variables is defined as $\mathbb E[XY]$ (?), I wonder if some geometrical intuition could be given via the triangle inequality. I have no idea how to proceed, but I was wondering if it makes any sense. $\endgroup$ – Antoni Parellada Jan 3 '17 at 19:21
  • 1
    $\begingroup$ @Antoni The triangle inequality is too general. An inner product is a much more special object. Fortunately, the appropriate geometrical intuition is precisely that of Euclidean geometry. Moreover, even in the case of random variables $X$ and $Y$, the necessary geometry can be confined to the two-dimensional real vector space generated by $X$ and $Y$: that is, to the Euclidean plane itself. In the present instance $X$ does not appear to be an RV: it's just an $n$-vector. Here, the space spanned by $X$ and $(1,1,\ldots, 1)$ is the Euclidean plane in which all the geometry happens. $\endgroup$ – whuber Jan 3 '17 at 19:52
  • 3
    $\begingroup$ Setting $\hat\beta_1=0$ in the reply I linked to, and dividing all terms by $n$ (if you wish) will give you the full algebraic solution for the variance: there's no reason to copy it all over again. That's because $\hat\beta_0$ is the arithmetic mean of $y$, whence $||y-\hat y||^2$ is just $n$ times the variance as you have defined it here, $||\hat y||^2$ is $n$ times the squared arithmetic mean, and $||y||^2$ is $n$ times the arithmetic mean of the squared values. $\endgroup$ – whuber Jan 3 '17 at 22:04
20
$\begingroup$

Expanding on @whuber's point in the comments, if $Y$ and $Z$ are orthogonal, you have the Pythagorean Theorem:

$$ \|Y\|^2 + \|Z\|^2 = \|Y + Z\|^2 $$

Observe that $\langle Y, Z \rangle \equiv \mathrm{E}[YZ]$ is a valid inner product and that $\|Y\| = \sqrt{\mathrm{E}[Y^2]}$ is the norm induced by that inner product.

Let $X$ be some random variable. Let $Y = \mathrm{E}[X]$, Let $Z = X - \mathrm{E}[X]$. If $Y$ and $Z$ are orthogonal:

\begin{align*} & \|Y\|^2 + \|Z\|^2 = \|Y + Z\|^2 \\ \Leftrightarrow \quad&\mathrm{E}[\mathrm{E}[X]^2] + \mathrm{E}[(X - \mathrm{E}[X])^2] = \mathrm{E}[X^2] \\ \Leftrightarrow \quad & \mathrm{E[X]}^2 + \mathrm{Var}[X]= \mathrm{E}[X^2] \end{align*}

And it's easy to show that $Y = \mathrm{E}[X]$ and $Z = X - \mathrm{E}[X]$ are orthogonal under this inner product:

$$\langle Y, Z \rangle = \mathrm{E}[\mathrm{E}[X]\left(X - \mathrm{E}[X] \right)] = \mathrm{E}[X]^2 - \mathrm{E}[X]^2 = 0$$

One of the legs of the triangle is $X - \mathrm{E}[X]$, the other leg is $\mathrm{E}[X]$, and the hypotenuse is $X$. And the Pythagorean theorem can be applied because a demeaned random variable is orthogonal to its mean.


Technical remark:

$Y$ in this example really should be the vector $Y = \mathrm{E}[X] \mathbf{1}$, that is, the scalar $\mathrm{E}[X]$ times the constant vector $\mathbf{1}$ (e.g. $\mathbf{1} = [1, 1, 1, \ldots, 1]'$ in the discrete, finite outcome case). $Y$ is the vector projection of $X$ onto the constant vector $\mathbf{1}$.

Simple Example

Consider the case where $X$ is a Bernoulli random variable where $p = .2$. We have:

$$ X = \begin{bmatrix} 1 \\ 0 \end{bmatrix} \quad P = \begin{bmatrix} .2 \\ .8 \end{bmatrix} \quad \mathrm{E}[X] = \sum_i P_iX_i = .2 $$

$$ Y = \mathrm{E}[X]\mathbf{1} = \begin{bmatrix} .2 \\ .2 \end{bmatrix} \quad Z = X - \mathrm{E}[X] = \begin{bmatrix} .8 \\ -.2 \end{bmatrix} $$

And the picture is: enter image description here

The squared magnitude of the red vector is the variance of $X$, the squared magnitude of the blue vector is $\mathrm{E}[X]^2$, and the squared magnitude of the yellow vector is $\mathrm{E}[X^2]$.

REMEMBER though that these magnitudes, the orthogonality etc... aren't with respect to the usual dot product $\sum_i Y_iZ_i$ but the inner product $\sum_i P_iY_iZ_i$. The magnitude of the yellow vector isn't 1, it is .2.

The red vector $Y = \mathrm{E}[X]$ and the blue vector $Z = X - \mathrm{E}[X]$ are perpendicular under the inner product $\sum_i P_i Y_i Z_i$ but they aren't perpendicular in the intro, high school geometry sense. Remember we're not using the usual dot product $\sum_i Y_i Z_i$ as the inner product!

$\endgroup$
  • $\begingroup$ That is really good! $\endgroup$ – Antoni Parellada Jan 5 '17 at 1:28
  • 1
    $\begingroup$ Good answer (+1), but it lacks a figure, and also might be a bit confusing for OP because your Z is their X... $\endgroup$ – amoeba Jan 5 '17 at 13:01
  • $\begingroup$ @MatthewGunn, great answer. you can check my answer below for a representation where orthogonality is in the Euclidean sense. $\endgroup$ – YBE Jan 5 '17 at 20:14
  • $\begingroup$ I hate to be obtuse, but I'm having trouble keeping $Z$, $Var(X)$, and the direction of the logic straight ('because' comes at places that don't make sense to me). It feels like a lot of (well substantiated) facts are stated randomly. What space is the inner product in? Why 1? $\endgroup$ – Mitch Jan 8 '17 at 17:10
  • $\begingroup$ @Mitch The logical order is: (1) Observe that a probability space defines a vector space; we can treat random variables as vectors. (2) Define the inner product of random variables $Y$ and $Z$ as $E[YZ]$. In an inner product space, vectors $Y$ and $Z$ are defined as orthogonal if their inner product is zero. (3a) Let $X$ be some random variable. (3b) Let $Y = E[X]$ and $Z = X - E[X]$. (4) Observe that $Y$ and $Z$ defined this way are orthogonal. (5) Since $Y$ and $Z$ are orthogonal, the pythagorean theorem applies (6) By simple algebra, the Pythagorean theorem is equivalent to the identity. $\endgroup$ – Matthew Gunn Jan 8 '17 at 18:10
6
$\begingroup$

I will go for a purely geometric approach for a very specific scenario. Let us consider a discrete valued random variable $X$ taking values $\{x_1,x_2\}$ with probabilities $(p_1,p_2)$. We will further assume that this random variable can be represented in $\mathbb{R}^2$ as a vector, $\mathbf{X} = \left(x_1\sqrt{p_1},x_2\sqrt{p_2} \right)$. enter image description here

Notice that the length-square of $\mathbf{X}$ is $x_1^2p_1+x_2^2p_2$ which is equal to $E[X^2]$. Thus, $\left\| \mathbf{X} \right\| = \sqrt{E[X^2]}$.

Since $p_1+p_2=1$, the tip of vector $\mathbf{X}$ actually traces an ellipse. This becomes easier to see if one reparametrizes $p_1$ and $p_2$ as $\cos^2(\theta)$ and $\sin^2(\theta)$. Hence, we have $\sqrt{p_1} =\cos(\theta)$ and $\sqrt{p_2} = \sin(\theta)$.

One way of drawing ellipses is via a mechanism called Trammel of Archimedes. As described in wiki: It consists of two shuttles which are confined ("trammelled") to perpendicular channels or rails, and a rod which is attached to the shuttles by pivots at fixed positions along the rod. As the shuttles move back and forth, each along its channel, the end of the rod moves in an elliptical path. This principle is illustrated in the figure below.

Now let us geometrically analyze one instance of this trammel when the vertical shuttle is at $A$ and the horizontal shuttle is at $B$ forming an angle of $\theta$. Due to construction, $\left|BX\right| = x_2$ and $\left| AB \right| = x_1-x_2$, $\forall \theta$ (here $x_1\geq x_2$ is assumed wlog).

enter image description here

Let us draw a line from origin, $OC$, that is perpendicular to the rod. One can show that $\left| OC \right|=(x_1-x_2) \sin(\theta) \cos(\theta)$. For this specific random variable \begin{eqnarray} Var(X) &=& (x_1^2p_1 +x_2^2p_2) - (x_1p_1+x_2p_2)^2 \\ &=& x_1^2p_1 +x_2^2p_2 - x_1^2p_1^2 - x_2^2p_2^2 - 2x_1x_2p_1p_2 \\ &=& x_1^2(p_1-p_1^2) + x_2^2(p_2-p_2^2) - 2x_1x_2p_1p_2 \\ &=& p_1p_2(x_1^2- 2x_1x_2 + x_2^2) \\ &=& \left[(x_1-x_2)\sqrt{p_1}\sqrt{p_2}\right]^2 = \left|OC \right|^2 \end{eqnarray} Therefore, the perpendicular distance $\left|OC \right|$ from the origin to the rod is actually equal to the standard deviation, $\sigma$.

If we compute the length of segment from $C$ to $X$: \begin{eqnarray} \left|CX\right| &=& x_2 + (x_1-x_2)\cos^2(\theta) \\ &=& x_1\cos^2(\theta) +x_2\sin^2(\theta) \\ &=& x_1p_1 + x_2p_2 = E[X] \end{eqnarray}

Applying the Pythagorean Theorem in the triangle OCX, we end up with \begin{equation} E[X^2] = Var(X) + E[X]^2. \end{equation}

To summarize, for a trammel that describes all possible discrete valued random variables taking values $\{x_1,x_2\}$, $\sqrt{E[X^2]}$ is the distance from the origin to the tip of the mechanism and the standard deviation $\sigma$ is the perpendicular distance to the rod.

Note: Notice that when $\theta$ is $0$ or $\pi/2$, $X$ is completely deterministic. When $\theta$ is $\pi/4$ we end up with maximum variance.

$\endgroup$
  • 1
    $\begingroup$ +1 Nice answer. And multiplying vectors by the square of the probabilities is a cool/useful trick to make the usual probabilistic notion of orthogonality look orthogonal! $\endgroup$ – Matthew Gunn Jan 5 '17 at 20:18
  • $\begingroup$ Great graphics. The symbols all make sense (the trammel describing an ellipse and then the Pythagorean Thm applies) but somehow I'm not getting intuitively how it gives an idea of how 'magically' it relates the moments (the spread and center. $\endgroup$ – Mitch Jan 8 '17 at 16:51
  • $\begingroup$ consider the trammel as a process that defines all the possible $(x_1,x_2)$ valued random variables. When the rod is horizontal or vertical you have a deterministic RV. In the middle there is randomness and it turns out that in my proposed geometric framework how random a RV (its std) is exactly measured by the distance of the rod to the origin. There might be a deeper relationship here as elliptic curves connects various objects in math but I am not a mathematician so I cannot really see that connection. $\endgroup$ – YBE Jan 8 '17 at 21:24
3
$\begingroup$

You can rearrange as follows:

$$ \begin{eqnarray} Var(X) &=& E[X^2] - (E[X])^2\\ E[X^2] &=& (E[X])^2 + Var(X) \end{eqnarray} $$

Then, interpret as follows: the expected square of a random variable is equal to the square of its mean plus the expected squared deviation from its mean.

$\endgroup$
  • $\begingroup$ Oh. Huh. Simple. But the squares still seem kinda uninterpreted. I mean it makes sense (sort of, extremely loosely) without the squares. $\endgroup$ – Mitch Jan 3 '17 at 19:39
  • 3
    $\begingroup$ I am not sold on this. $\endgroup$ – Michael Chernick Jan 3 '17 at 19:57
  • 1
    $\begingroup$ If the Pythagorean theorem applies, what is the triangle with what sides and how are the two legs perpendicular? $\endgroup$ – Mitch Jan 4 '17 at 15:31
1
$\begingroup$

Sorry for not having the skill to elaborate and provide a proper answer, but I think the answer lies in the physical classical mechanics concept of moments, especially the conversion between 0 centred "raw" moments and mean centred central moments. Bear in mind that variance is the second order central moment of a random variable.

$\endgroup$
1
$\begingroup$

The general intuition is that you can relate these moments using the Pythagorean Theorem (PT) in a suitably defined vector space, by showing that two of the moments are perpendicular and the third is the hypotenuse. The only algebra needed is to show that the two legs are indeed orthogonal.

For the sake of the following I'll assume you meant sample means and variances for computation purposes rather than moments for full distributions. That is:

$$ \begin{array}{rcll} E[X] &=& \frac{1}{n}\sum x_i,& \rm{mean, first\ central\ sample\ moment}\\ E[X^2] &=& \frac{1}{n}\sum x^2_i,& \rm{second\ sample\ moment\ (non-central)}\\ Var(X) &=& \frac{1}{n}\sum (x_i - E[X])^2,& \rm{variance, second\ central\ sample\ moment} \end{array} $$

(where all sums are over $n$ items).

For reference, the elementary proof of $Var(X) = E[X^2] - E[X]^2$ is just symbol pushing: $$ \begin{eqnarray} Var(X) &=& \frac{1}{n}\sum (x_i - E[X])^2\\ &=& \frac{1}{n}\sum (x^2_i - 2 E[X]x_i + E[X]^2)\\ &=& \frac{1}{n}\sum x^2_i - \frac{2}{n} E[X] \sum x_i + \frac{1}{n}\sum E[X]^2\\ &=& E[X^2] - 2 E[X]^2 + \frac{1}{n} n E[X]^2\\ &=& E[X^2] - E[X]^2\\ \end{eqnarray} $$

There's little meaning here, just elementary manipulation of algebra. One might notice that $E[X]$ is a constant inside the summation, but that is about it.

Now in the vector space/geometrical interpretation/intuition, what we'll show is the slightly rearranged equation that corresponds to PT, that

$$ \begin{eqnarray} Var(X) + E[X]^2 &=& E[X^2] \end{eqnarray} $$

So consider $X$, the sample of $n$ items, as a vector in $\mathbb{R}^n$. And let's create two vectors $E[X]{\bf 1}$ and $X-E[X]{\bf 1}$.

The vector $E[X]{\bf 1}$ has the mean of the sample as every one of its coordinates.

The vector $X-E[X]{\bf 1}$ is $\langle x_1-E[X], \dots, x_n-E[X]\rangle$.

These two vectors are perpendicular because the dot product of the two vectors turns out to be 0: $$ \begin{eqnarray} E[X]{\bf 1}\cdot(X-E[X]{\bf 1}) &=& \sum E[X](x_i-E[X])\\ &=& \sum (E[X]x_i-E[X]^2)\\ &=& E[X]\sum x_i - \sum E[X]^2\\ &=& n E[X]E[X] - n E[X]^2\\ &=& 0\\ \end{eqnarray} $$

So the two vectors are perpendicular which means they are the two legs of a right triangle.

Then by PT (which holds in $\mathbb{R}^n$), the sum of the squares of the lengths of the two legs equals the square of the hypotenuse.

By the same algebra used in the boring algebraic proof at the top, we showed that we get that $E[X^2]$ is the square of the hypotenuse vector:

$(X-E[X])^2 + E[X]^2 = ... = E[X^2]$ where squaring is the dot product (and it's really $E[x]{\bf 1}$ and $(X-E[X])^2$ is $Var(X)$.

The interesting part about this interpretation is the conversion from a sample of $n$ items from a univariate distribution to a vector space of $n$ dimensions. This is similar to $n$ bivariate samples being interpreted as really two samples in $n$ variables.

In one sense that is enough, the right triangle from vectors and $E[X^2]$ pops out as the hypotnenuse. We gave an interpretation (vectors) for these values and show they correspond. That's cool enough, but unenlightening either statistically or geometrically. It wouldn't really say why and would be a lot of extra conceptual machinery to, in the end mostly, reproduce the purely algebraic proof we already had at the beginning.

Another interesting part is that the mean and variance, though they intuitively measure center and spread in one dimension, are orthogonal in $n$ dimensions. What does that mean, that they're orthogonal? I don't know! Are there other moments that are orthogonal? Is there a larger system of relations that includes this orthogonality? central moments vs non-central moments? I don't know!

$\endgroup$
  • $\begingroup$ I am also interested in an interpretation/intuition behind the superficially similar bias variance tradeoff equation. Does anybody have hints there? $\endgroup$ – Mitch Jan 12 '17 at 17:08
  • $\begingroup$ Let $p_i$ be the probability of state $i$ occurring. If $p_i = \frac{1}{n}$ then $\sum_i p_i X_i Y_i = \frac{1}{n} \sum_i X_i Y_i$, that is, $E[XY]$ is simply the dot product between $X$ and $Y$ divided by $n$. If $\forall_i p_i = \frac{1}{n}$, what I used as an inner product ( $E[XY] = \sum_i p_i X_i Y_i$) is basically the dot product divided by $n$. This whole Pythagorean interpretation still needs to you use the particular inner product $E[XY]$ (though it's algebriacly close to the classic dot product for a probability measure $P$ such that $\forall_i p_i = \frac{1}{n}$). $\endgroup$ – Matthew Gunn Jan 12 '17 at 18:01
  • $\begingroup$ Btw, the trick @YBE did is to define new vectors $\hat{x}$ and $\hat{y}$ such that $\hat{x}_i = x_i \sqrt{p_i}$ and $\hat{y}_i = x_i \sqrt{p_i}$. Then dot product $\hat{x} \cdot \hat{y} = \sum_i x_i \sqrt{p_i} y_i \sqrt{p_i} = \sum_i p_i x_i y_i = E[xy]$.The dot product of $\hat{x}$ and $\hat{y}$ corresponds to $E[xy]$ (which is what I used as an inner product). $\endgroup$ – Matthew Gunn Jan 12 '17 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.