4
$\begingroup$

Suppose that $x_1,\ldots,x_n$ are real numbers, and let $\bar x$ denote the average $\frac{\sum_i x_i}n.$ I know how to prove on paper that the equality $$\sum_i (x_i - \bar x)^2 = \sum_i (x_i - \bar x) x_i$$ holds true in general, but I do not have any intuition for why this should be the case. Similarly, I do not know how to rationalize with geometric intuition the identity $$\sum_i (x_i - \bar x) y_i=\sum_i (x_i - \bar x)(y_i - \bar y) = \sum_ix_i (y_i - \bar y).$$ Do you have a way of visualizing what is going on in these equalities (e.g. in terms of weighted sums of data points)?

$\endgroup$
  • $\begingroup$ SInce we're not taking about physics but numbers (and often, numbers representing a non-physical quantity), what is the intent of the word "physical" in your question? What would distinguish physical intuition from non-physical intuition? What makes a "weighted sum" necessarily physical, for example? I'm simply unclear about what kinds of explanations this adjective is intended to include or exclude. $\endgroup$ – Glen_b Oct 10 '17 at 23:50
  • $\begingroup$ The simplest explanation is in the expansion $(x_i-\bar{x})^2=x_i^2-2x_i\bar{x}+\bar{x}^2$, which upon summing will simplify the last two terms. $\endgroup$ – Alex R. Oct 11 '17 at 0:02
  • 1
    $\begingroup$ I don't know, but here's an interesting way of looking at it: suppose we define vectors $\mathbf{x} = \{x_i\}$, $\mathbf{x}' = \mathbf{x} - \overline{x} \mathbf{J}$ with $\mathbf{J} = (1, 1, ..., 1)$. Then the last equation can be written as a dot product identity $\mathbf{x}' \cdot \mathbf{y}' = \mathbf{x}' \cdot \mathbf{y} = \mathbf{x} \cdot \mathbf{y}' = \mathbf{x} \cdot \mathbf{y}$. I never knew that about vectors before. $\endgroup$ – Dave Kielpinski Oct 11 '17 at 0:06
  • $\begingroup$ @Glen_b I am particularly interested in getting some visual intuition for this. I said "physical" because (in my experience, at least) intuition concerning the physical world (or concerning physics) is easily visualized. One physics example that comes to mind is the Moment of Inertia, which is proportional to $\sum_i (x_i - \bar x) ^2 $ (where each $x_i$ is the position of a point-mass in a rigid system). $\endgroup$ – Jasha Oct 11 '17 at 0:13
  • $\begingroup$ @DaveKielpinski I'm not sure if it's true that $\mathbf x' \cdot \mathbf y'$ is equal to $\mathbf x \cdot \mathbf y$, which would seem to suggest that $\sum_i (x_i - \bar x) (y_i - \bar y)$ is equal to $\sum_i x_i y_i$. $\endgroup$ – Jasha Oct 11 '17 at 0:22
3
$\begingroup$

This is the geometry of (orthogonal) projections and Pythagoras' theorem. Let $J$ be the all 1's $n\times n$ matrix, note that $J^2=nJ$. Now the centering operation $x_i \to x_i-\bar{x}$ is the operator $\text{Id}-\text{Average}$ which can be represented by the matrix $H=I-n^{-1} J$. $H$ is a projection matrix called the centering matrix ($H^T=H, H^2=H$).

Now $$ \sum (x_i-\bar{x})^2 = \lVert Hx \rVert^2 = x^T H x \quad\text{and}\\ \sum (x_i-\bar{x}) x_i = (Hx)^T x =(Hx)^T (x-Hx+Hx)=0+x^T Hx $$ since by general properties of projections, $(x-Hx) \perp Hx$ so the dot product is zero.

So this is simply an expression of the general geometric fact that the dot product of a projected vector by itself equals the dot product of the vector by its projection. Just draw a diagram in the plane, since there is only two vectors $x$ and $Hx$ involved, all the real actions occurs in the plane generated by those two vectors:

illustration of dot product of projection with unprojected vector

(Image contributed by @Martijn Weterings)

$\endgroup$
  • 2
    $\begingroup$ So in short: (1): $(X-\bar{X})$ is a projection of the point $X$ (onto the plane perpendicular to the line $x_1=...=x_n=c$ and trough the point $x_i=...=x_n=\bar{x}$) (2): the dot product of a projection with itself is equal to the dot product of the projection with the unprojected point. $\endgroup$ – Martijn Weterings Jan 18 at 9:04
  • 1
    $\begingroup$ Here is an image i.stack.imgur.com/sYZpy.png that may possibly be used to illustrate the principle. $\endgroup$ – Martijn Weterings Jan 18 at 9:27
  • $\begingroup$ I see. So one can use the same argument to derive $n|\bar x|^2=\bar x\sum x_i$, using $n^{-1}J$ as the projection matrix instead of $H$. $\endgroup$ – Jasha Jan 18 at 16:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.