2
$\begingroup$

When doing principal components it is intuitively clear that noise accumulates more towards the last components. What would be the formal explanation of how and why this happens?

$\endgroup$
1
  • 3
    $\begingroup$ A formal explanation requires a formal statement of the assumptions of an underlying probability model. Typically PCA is used without such assumptions - a mere transformation. See here for some examples of the danger of interpreting smaller components as noise. $\endgroup$ – Scortchi - Reinstate Monica Mar 3 '14 at 10:11
5
$\begingroup$

It is not "intuitively clear" to me. In fact, I'd say it totally depends on the signal to noise ratio. PCA decomposes the data into uncorrelated vectors, with the first vector (first component) catching the largest portion of the variability in the data, the second -- the second largest etc.

If the noise is big enough, it will push the "signal" towards the smaller components.

Quite often I see that one of the "major" components is dominated by one or two outliers (usually technical errors in measurement), while the actual study variables come in third or fourth.

$\endgroup$
1
  • 1
    $\begingroup$ Can you have much Signal without variability? Would'nt it be like an independent variable with a Slope close to zero. $\endgroup$ – Sympa Mar 3 '14 at 20:52
1
$\begingroup$

Maybe another way to state the same thing is that the first Principal Component grabs the majority of the Signal. The 2nd Principal Component is stuck attempting to explain only what is unexplained by the 1st one. It explains the residual of the 1st one. By the time you reach the 3d Principal Component it really does not have much Signal left to work with or explain because the vast majority of it has been explained by the first two. This logic entails that as you move away from the 1st Principal Component and towards the last one, you get less and less Signal... which means you get more and more Noise.

$\endgroup$
3
  • 6
    $\begingroup$ ... assuming that most of the variability in your data consists of signal. If most of your data is noise, then the first few components will mostly represent that noise, and not the signal. $\endgroup$ – fabians Mar 3 '14 at 8:48
  • 1
    $\begingroup$ ... that's true. Unpredictive Data = Weak PCA. But, on a relative basis is there any reason to believe that there could possibly be more Signal in the 3d Principal Component as in the 1st one (as another commentator suggests)? $\endgroup$ – Sympa Mar 3 '14 at 20:55
  • 2
    $\begingroup$ I think the confusion is about what constitues a "signal" -- if you define "signal" as "major modes of variation in the data" then PCA by definition captures those in the first few components. If "signal" is defined to mean something like "underlying structure of the data" or "components of the data useful for predicting some external quantity" then -- if the data contains lots of noise -- that kind of "signal" won't show up in the first few components, or at least it will be heavily contaminated with noise. See example here: rpubs.com/fabian-s/13957 $\endgroup$ – fabians Mar 4 '14 at 10:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.