3
$\begingroup$

I was reading up on Non-linear dimensionality reduction. On the Wikipedia article it says that PCA does not preserve the intrinsic geometry of the data. What kind of information is it preserving?

It is interesting, because we are currently working on a problem dealing with supervised learning on images. We don't know what model best suits it, yet. If applying PCA doesn't take into consideration the intrinsic geometry, then is it worthwhile to apply PCA to the data before passing it to a classifier? For instance, the images contain overlapping objects that might be crucial to the classifier. But an argument could be made that we can use PCA to find a basis that has lower noise (in the sense that we may believe that noise correlates with large patches of pixels).

$\endgroup$

1 Answer 1

1
$\begingroup$

PCA forms linear combinations of the variables that preserve as much of the variance as possible, subject to orthogonality.

That is, the first PC preserves as much of the variance as possible. The second preserves as much as possible, given that it is orthogonal (uncorrelated) with the first. The third has to be orthogonal to both of the first two. And so on.

$\endgroup$
6
  • $\begingroup$ I think knew about the variance part. It is said in the link, that NLDR can represent intrinsic variables like "scale" and "rotation" (for their example with the letter A). I guess my question is that how exactly I can think of the variables I get? Can it still be stuff like "rotation" and "scale" as long as they are linearly varying? Are "rotation" and "scale" linear? How can I find out what kind of variables PCA or NLDR is capturing? $\endgroup$
    – learning
    Commented Jun 26, 2017 at 4:38
  • $\begingroup$ Eg., in the case I provided in my question, can "overlap" be a variable being captured? $\endgroup$
    – learning
    Commented Jun 26, 2017 at 4:40
  • $\begingroup$ The variables that you get out of PCA are linear combinations of the variables that went into it. So, it would be something like "scale * 0.4 + rotation *0.3" and so on. $\endgroup$
    – Peter Flom
    Commented Jun 26, 2017 at 12:03
  • $\begingroup$ Ahh, thanks!, I guess my final question would be to ask if we can somehow determine if the variables vary non-linearly or linearly. $\endgroup$
    – learning
    Commented Jun 26, 2017 at 17:10
  • $\begingroup$ That doesn't make sense. "Linear" is an adjective that applies to relationships, not variables. $\endgroup$
    – Peter Flom
    Commented Jun 27, 2017 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.