Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 3 down vote favorite

I have a quick question that I am having trouble wrapping my head around. The PCA algorithm can be formulated in terms of the correlation matrix (assume the data $X$ has already been normalized and we are only considering projection onto the first PCA). The object function can be written as:

$$ \max_w (Xw)^T(Xw)\; \: \text{s.t.}$$ $$ w^Tw = 1 $$.

This is fine, and we use Lagrangian multipliers to solve it. But this can be rewritten as:

$$ \max_w (Xw)^T(Xw) - \lambda w^Tw $$

which is equivilant to

$$ \max_w \frac{ (Xw)^T(Xw) }{w^Tw} $$

$$= \max_w \sum_{i=1}^n \text{ (distance from point $x_i$ to line $w$ ) }^2 $$ see here

But this is saying to maximize the distance between point and line, and from what I've read here, this is incorrect ( it should be $\min$, not $\max$. Where is my error? Or, can someone show me the link between maximizing variance in projected space and minimizing distance between point and line?

share|improve this question
I think minimum distance is used to meet the criterion of orthogonality for the components. The points are projected into the PCs that are orthogonal to each other but in each successive component the remaining variance is maximized. – Michael Chernick Jul 12 '12 at 15:42
Hint: What happens when you consider the smallest eigenvalue first, rather than the largest one? – whuber♦ Jul 12 '12 at 15:45
@whuber The smallest eigenvalue probably has the PC that is the solution to the final objective function. But this PC does not maximixe the original objective function. – Cam.Davidson.Pilon Jul 12 '12 at 17:06
1  
I'm not sure what you mean by "final" and "original" objective function, Cam. PCA is not (conceptually) an optimization program. Its output is a set of principal directions, not just one. It is an (interesting) mathematical theorem that these directions can be found by solving a sequence of constrained quadratic programs, but that's not basic to the concepts or the practice of PCA. I am only suggesting that, by focusing on the smallest eigenvalue rather than on the largest one, you can reconcile the two ideas of (1) minimizing distances and (2) taking an optimization view of PCA. – whuber♦ Jul 12 '12 at 18:32
@whuber I was referring to the first and last opt. objectives in my post (I shouldn't have wrote 'function'). Perhaps I should rephrase it out of the PCA context (though that is my final application): can one connect the two optimization objectives, the first being maximize variance, and the second being minimize distances to line. – Cam.Davidson.Pilon Jul 12 '12 at 19:22
show 1 more comment

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.