I think that everyone starts explaining PCA from the wrong end: from eigenvectors. My answer starts at the right place: coordinate system. Eigenvectors, and eigenproblem in general, are the mathematical tool that is used to address the real issue at hand which is a wrong coordinate system. I'll explain.

Let's start with a line. What is a line? It's a one dimensional object. So, you need only one dimension to move from one point to another. On a plane though you attach two coordinate any point of a line. That is because with respect to a line itself the coordinate system is chosen arbitrarily. The coordinate system, I would argue, does not reflect the inner one dimensional nature of the line. If only I would always put the origin of my Cartesian coordinate system on the line, and turned it so that its x-axis was on the line, then I would not need y-axis anymore! All my points are on one axis, because a line is a one dimensional object.

That's where PCA explanations should start. The eigen problem is a tool that does the rotation which I described, and de-meaning of variables puts the origin onto the line. PCA helps reveal true dimensions of the data so long the relationships between the variables are linear.

I think that everyone starts explaining PCA from the wrong end: from eigenvectors. My answer starts at the right place: coordinate system. Eigenvectors, and eigenproblem in general, are the mathematical tool that is used to address the real issue at hand which is a wrong coordinate system. I'll explain.

Let's start with a line. What is a line? It's a one dimensional object. So, you need only one dimension to move from one point to another. On a plane though you attach two coordinates to any point of a line.

That is because with respect to a line itself the coordinate system is chosen arbitrarily. Take a look at this line.

Does it look like a different object than the previous line? It may, if you keep looking at the coordinate. However, if you forget about the coordinate system, and just look at it as a geometrical object in space, then these two lines are identical!

The coordinate system, I would argue, does not reflect the inner one dimensional nature of the line. If only I would always put the origin of my Cartesian coordinate system on the line, and turned it so that its x-axis was on the line, then I would not need y-axis anymore! All my points are on one axis, because a line is a one dimensional object.

That's where PCA explanations should start. The eigen problem is a tool that does the rotation which I described, plus de-meaning of variables puts the origin onto the line. PCA helps reveal true dimensions of the data so long the relationships between the variables are linear.