6
$\begingroup$

I can't seem to understand how PCA works. My (lack of) math knowledge don't help me either. I have read that the new set of variables has to be a linear combination of the old set.

What does that mean exactly? That there should be a way to multiply numbers a, b, c, ... with dimensions/variables x, y, z... and get the old bigger(!) set?

Can you answer my example: if I have 8 variables/dimension can they be reduced to 3? Or a vector of 3 components (sorry for my lack of proper terminolgy, English is not my native language) is not a linear combination of 8 and thus no?

Thanks.

$\endgroup$
7
$\begingroup$

+1 for ttnphns, but I'll try to give tl;dr, math free version.

Your doubts are fully justified -- one cannot stuff 8 dims as <8 linear combinations in a general case. What PCA really does is that it converts 8 dims into 8 linear combinations in such a way that it stuffs how much diversity of the data possible to the first dim, than stuffs the most of remains in the second one and so on -- thus one may expect that the last dims contain only noise coming from errors and noises in the original data and may be omitted, what leads to a reduction of dimensionality.

This way one can imagine it as lossy compression algorithm like MP3 or JPEG -- it dumps some of the original information, but hopefully only this that doesn't matter.

$\endgroup$
  • $\begingroup$ +1 and thanks, But :-) what is "tl;dr" for me or Ion, both not native English speakers? $\endgroup$ – ttnphns Jan 30 '12 at 10:22
  • $\begingroup$ @ttnphns I'm not a native English speaker too, so I'll use a link to an authoritative source. Tl;dr it means too long; didn't read. $\endgroup$ – user88 Jan 30 '12 at 11:54
7
$\begingroup$

Ion, PCA is just a specific case of orthogonal rotation. Let X be your n x p data matrix of n points in p dimensions (axes). To obtain this same cloud of points in a new set of axes somehow rotated in space relatively the old ones, you multiply X by a p x p matrix Q of cosines between the old axes (rows) and new axes (columns): $\bf{XQ=C}$ [1], where C is your new (rotated) coordinates. This formula says that each new dimension is a linear combination of p old dimensions. It also follows that $\bf {X=CQ^{-1}}$ or, since the rotation was orthogonal - $\bf Q$ is orthonormal matrix, - $\bf {X=CQ'}$ [2] which says that each old dimension is a linear combination of p new dimensions.

Now, PCA is virtually this rotation; what makes PCA special is that Q is not an arbitrary rotation matrix; it is the matrix of such a rotation so that the sum-of-squares (or variance, if your data had been centered) in the 1st column of C becomes maximal possible: that is, variability along the 1st principal component is maximized. Then, sum-of-squares of in the 2nd C column (2nd principal component) is second maximal. Etc. Each next component is a new axis which takes off less and less of multidimensional variability in the cloud. Hence, lion's share of the variability is accounted for by only few m (m<p) new axes (principal components).

In PCA, Q is called matrix of eigenvectors (these being its columns). If you retain just m first components, by retaining just m first columns in Q, you still can use formula [1] to obtain component scores for the m components - the points' coordinates on these m dimensions. So, whatever is m, each component remains a linear combination of original variables. However, using then formula [2] to obtain p original variables from m components won't give you original variables exactly: each original variable will be a linear combination of m components plus some error term. If you perform linear regression (without constant term) of each original variable by m components as predictors you will see that regression coefficients you get are the elements of Q.

$\endgroup$
5
$\begingroup$

Here's my stab at this; completely without math, just some basic principles and a picture. You asked for it. ;)

Consider the scenario in the picture below. You have 2D data points along the X and Y axis. You could use the PCA to find the principal axis P.

PCA

The point of this analysis is that if your data are distributed this way, you don't really need both X and Y to work with them. You might as well only use one dimension, along P.

If you have N-dimensional input space, you can use the PCA to reduce it to anywhere between 1 to N dimensions. So yes, you can reduce from 8 to 3; whether that makes any sense to do is up to your decision (based upon the concrete data in question).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.