Suppose that $x_1,\ldots,x_n$ are real numbers, and let $\bar x$ denote the average $\frac{\sum_i x_i}n.$ I know how to prove on paper that the equality $$\sum_i (x_i - \bar x)^2 = \sum_i (x_i - \bar x) x_i$$ holds true in general, but I do not have any intuition for why this should be the case. Similarly, I do not know how to rationalize with geometric intuition the identity $$\sum_i (x_i - \bar x) y_i=\sum_i (x_i - \bar x)(y_i - \bar y) = \sum_ix_i (y_i - \bar y).$$ Do you have a way of visualizing what is going on in these equalities (e.g. in terms of weighted sums of data points)?
9
-
$\begingroup$ SInce we're not taking about physics but numbers (and often, numbers representing a non-physical quantity), what is the intent of the word "physical" in your question? What would distinguish physical intuition from non-physical intuition? What makes a "weighted sum" necessarily physical, for example? I'm simply unclear about what kinds of explanations this adjective is intended to include or exclude. $\endgroup$– Glen_bCommented Oct 10, 2017 at 23:50
-
$\begingroup$ The simplest explanation is in the expansion $(x_i-\bar{x})^2=x_i^2-2x_i\bar{x}+\bar{x}^2$, which upon summing will simplify the last two terms. $\endgroup$– Alex R.Commented Oct 11, 2017 at 0:02
-
1$\begingroup$ I don't know, but here's an interesting way of looking at it: suppose we define vectors $\mathbf{x} = \{x_i\}$, $\mathbf{x}' = \mathbf{x} - \overline{x} \mathbf{J}$ with $\mathbf{J} = (1, 1, ..., 1)$. Then the last equation can be written as a dot product identity $\mathbf{x}' \cdot \mathbf{y}' = \mathbf{x}' \cdot \mathbf{y} = \mathbf{x} \cdot \mathbf{y}' = \mathbf{x} \cdot \mathbf{y}$. I never knew that about vectors before. $\endgroup$– Dave KielpinskiCommented Oct 11, 2017 at 0:06
-
$\begingroup$ @Glen_b I am particularly interested in getting some visual intuition for this. I said "physical" because (in my experience, at least) intuition concerning the physical world (or concerning physics) is easily visualized. One physics example that comes to mind is the Moment of Inertia, which is proportional to $\sum_i (x_i - \bar x) ^2 $ (where each $x_i$ is the position of a point-mass in a rigid system). $\endgroup$– JashaCommented Oct 11, 2017 at 0:13
-
$\begingroup$ @DaveKielpinski I'm not sure if it's true that $\mathbf x' \cdot \mathbf y'$ is equal to $\mathbf x \cdot \mathbf y$, which would seem to suggest that $\sum_i (x_i - \bar x) (y_i - \bar y)$ is equal to $\sum_i x_i y_i$. $\endgroup$– JashaCommented Oct 11, 2017 at 0:22
|
Show 4 more comments
1 Answer
$\begingroup$
$\endgroup$
3
This is the geometry of (orthogonal) projections and Pythagoras' theorem. Let $J$ be the all 1's $n\times n$ matrix, note that $J^2=nJ$. Now the centering operation $x_i \to x_i-\bar{x}$ is the operator $\text{Id}-\text{Average}$ which can be represented by the matrix $H=I-n^{-1} J$. $H$ is a projection matrix called the centering matrix ($H^T=H, H^2=H$).
Now $$ \sum (x_i-\bar{x})^2 = \lVert Hx \rVert^2 = x^T H x \quad\text{and}\\ \sum (x_i-\bar{x}) x_i = (Hx)^T x =(Hx)^T (x-Hx+Hx)=0+x^T Hx $$ since by general properties of projections, $(x-Hx) \perp Hx$ so the dot product is zero.
So this is simply an expression of the general geometric fact that the dot product of a projected vector by itself equals the dot product of the vector by its projection. Just draw a diagram in the plane, since there is only two vectors $x$ and $Hx$ involved, all the real actions occurs in the plane generated by those two vectors:
(Image contributed by @Martijn Weterings)
answered Jan 17, 2019 at 10:58
-
2$\begingroup$ So in short: (1): $(X-\bar{X})$ is a projection of the point $X$ (onto the plane perpendicular to the line $x_1=...=x_n=c$ and trough the point $x_i=...=x_n=\bar{x}$) (2): the dot product of a projection with itself is equal to the dot product of the projection with the unprojected point. $\endgroup$ Commented Jan 18, 2019 at 9:04
-
1$\begingroup$ Here is an image i.sstatic.net/sYZpy.png that may possibly be used to illustrate the principle. $\endgroup$ Commented Jan 18, 2019 at 9:27
-
$\begingroup$ I see. So one can use the same argument to derive $n|\bar x|^2=\bar x\sum x_i$, using $n^{-1}J$ as the projection matrix instead of $H$. $\endgroup$– JashaCommented Jan 18, 2019 at 16:07
