7
$\begingroup$

What is the first principal component of points that form a "filled" rectangle in the 2D space?

Is it one of the diagonals? Or are the first two principal components basically the sides of the rectangle?

$\endgroup$
  • $\begingroup$ I edited your question, assuming you meant 2D. Please edit again if I misunderstood. Good question, by the way, +1. $\endgroup$ – amoeba Feb 8 '15 at 14:27
  • $\begingroup$ I meant 3D space, since the example I had in mind was a 3D point cloud, but as far as I know it should not matter. $\endgroup$ – hatero Feb 8 '15 at 15:34
  • $\begingroup$ In 3D the shape would probably be called "rectangular parallelepiped". "Rectangle" is a flat 2D shape, hence my confusion. But you are right: in this case what holds for 2D, holds for 3D as well. $\endgroup$ – amoeba Feb 8 '15 at 16:03
9
$\begingroup$

Imagine data points filling a 2D rectangle in the center of the coordinate system, with its sides oriented along the coordinate axes: from $-a$ to $a$ along the $x$-axis, and from $-b$ to $b$ along the $y$-axis.

The projection on $x$ is a uniform distribution with variance $a^2/3$. The projection on $y$ is also a uniform distribution with variance $b^2/3$. Since $x$ and $y$ are obviously not correlated (if this is not obvious, ask yourself whether the correlation should be positive or negative?.. due to symmetry it can only be zero), the covariance between them is zero. This yields the covariance matrix $$\left(\begin{array}{c}a^2/3&0\\0&b^2/3\end{array}\right).$$ The task of PCA is to diagonalize the covariance matrix. But this one is already diagonal! This means that no rotation is necessary, and $x$-axis and $y$-axis are themselves principal axes. If e.g. $a>b$, then the $x$-axis is the first PC.

This might be a bit counter-intuitive: it might seem that a projection on the diagonal should have larger variance than the projection on the longer side; but it is in fact not so.


Bonus: Dzhanibekov effect

You seem to have meant a 3D rectangular parallelepiped instead of 2D rectangle. The arguments of course stay the same: covariance matrix is $3\times 3$ but still diagonal with principal axes being the coordinate axes.

Incidentally, there is a curious effect in mechanics concerning rotating solid body with three different moments of inertia (which is a mechanics analog of variance). It turns out that rotations around the axes with the largest and the smallest moment of inertia are stable, but rotation around the axis with the middle moment of inertia is unstable. Moreover, a rotating body will experience sudden "flips", which is known as Dzhanibekov effect -- after a Russian cosmonaut who observed it in space. One can easily observe it when spinning a book or a table tennis racket. See the following great threads on mathoverflow and on physics.SE and these videos:

| cite | improve this answer | |
$\endgroup$
3
$\begingroup$

amoeba's (great) answer says:

This might be a bit counter-intuitive: it might seem that a projection on the diagonal should have larger variance than the projection on the longer side; but it is in fact not so.

Indeed it is counter-intuitive, but maybe we can counter the counter-intuition?
(As intuition is my goal here, I won't try to be rigorous. Beware.)

For that, let's look at a rectangle which is oriented along the axes, with $a=80$ and $b=20$ (sticking to amoeba's notations). In the first images, the black lines are the directions of the original basis, and the green lines - of the new basis. Images on the left show the world according to the original basis, and those on the right - according to the new basis.

A Warm Up: Is the new covariance matrix diagonal?

Recall that $\text{cov}(X,Y)=E[(X-E[X])(Y-E[Y])]$.
The center of the data is the origin (both before and after the change of basis), so $E[X]=0=E[Y]$ and $\text{cov}(X,Y)=E[XY]$. i.e. data points in the first and third quadrants "push" the value of the covariance up, while data points in the second and fourth quadrants "push" the value of the covariance down.
Looking at the image, you can see that:

  • In the original basis, for every data point that "pushes up" (purple), there is a data point that "pushes down" (orange) with the same "force" (i.e. $|XY|$ is equal for both), so that they cancel each other's efforts. Therefore, we get $\text{cov}(X,Y)=0$.
  • In the new basis, there are much more data points that "push down", and many of them with more "force" (i.e. $|XY|$ is higher). Therefore, we get $\text{cov}(X,Y)<0$.

covariance demonstration So in the new basis, the covariance matrix is of the shape $\left(\begin{array}{c}+&-\\-&+\end{array}\right)$, i.e. not diagonal. Good.

By the way, in case of a square, the covariance matrix is diagonal before and after the change of basis: covariance demonstration for a square

The Real Deal: Does the projection on the diagonal have a higher variance?

Recall that $\text{Var}(X)=E[(X-E[X])^2]$. As aforementioned, $E[X]=0$, and thus $\text{Var}(X)=E[X^2]$.
So in our case, the variance along the horizontal axis is a measure of the horizontal distances between data points and the origin.

To start building our intuition, let's check for how many data points $x > 70$, i.e. their horizontal distances from the origin are higher than $70$:

  • In the original basis - $33$ data points
  • In the new basis - $26$ data points

data points with x > 70 Looks good, but I am not convinced yet that on average the horizontal distances from the origin decrease after the change of basis.

Let's try another approach: examine the difference in the X value of each data point (caused by the change of basis):

difference in X value

So, what did the change of basis do to the horizontal distances from the origin?

  • increased them by more than $5$ for $0$ data points
  • increased them by around $4$ for $10.6\%$ of the data points
  • increased them by around $2$ for $18.6\%$ of the data points
  • left them quite the same for $19.3\%$ of the data points
  • decreased them by around $2$ for $17.7\%$ of the data points
  • decreased them by around $4$ for $19.1\%$ of the data points
  • decreased them by more than $5$ for $14.6\%$ of the data points

And more roughly:

  • increased them for $39.5\%$ of the data points
  • left it the same for $1$ data point (the origin)
  • decreased them for $60.3\%$ of the data points

It is quite obvious that on average the horizontal distances decreased after the change of basis (i.e. the variance along the horizontal axis is lower after the change of basis!), but for me, the intuition is still hard to grasp.

Let's try to look at the rough version of the same image:

sign of difference in X value

Much better.
I still find it hard to articulate the intuition (mainly as I am still struggling with it myself), but I hope that the images speak for themselves.

Finally, for completeness' sake, the same image, but in the case of a square:

sign of difference in X value for a square

(The horizontal distance from the origin increased/decreased for exactly $50\%$ of the data points, and the variance along the horizontal axis is the same before and after the change of basis.)

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.