1
$\begingroup$

I read a paper detailing the algebraic process of kernel PCA. I have question though: the paper details the projection of new points onto the new eigenvectors in the feature space, but what if I want eigenvectors themselves? I am performing an analysis that compares eigenvector magnitudes in a Gaussian inner product space.

The closest I found in the paper was the definition of the eigenvectors as a hypothetical linear combination of data vectors in the kernel space:

$$\mathbf{V} = \sum_{i = 1}^l \alpha_i \Phi\left(\mathbf{x}_i\right).$$

From the paper, it seems that you derive all such $\alpha$ with the relation

$$\ell\lambda\mathbf{\alpha}=K\mathbf{\alpha}$$ (where $\mathbf{\alpha}$ is supposed to look bold as a vector).

But even after you obtain $\alpha_i$, you can't compute the eigenvectors in the feature space from the previous formula because, in the case of a Gaussian kernel that operates on $\|\mathbf{x} - \mathbf{y}\|$, there are infinitely many dimensions in the the "hidden" range of $\Phi$.

How can I compute $\mathbf{V}$, the eigenvectors in the $\mathbf{\Phi}$?


Edit 0

I understand that we can't formalize eigenvectors in a infinite-dimensional space implied by a Gaussian-norm inner product kernel. What I actually want is the length of the eigenvector. In the linear case, that would be the semimajor axes of the imaginary ellipse. I want to be able to have, for a "classification" PCA case as this one, a graph like the one on the left showing a curved (for lack of a better word) eigenvector projected down to the input space. I'd prefer a parametric form over which I may compute an arclength integral.


Edit 1

Oops. I meant principal component, not eigenvector.

$\endgroup$
  • 3
    $\begingroup$ Depending on the kernel, feature space can be very large; for example, for the Gaussian kernel the feature space is infinite-dimensional. How would you want to "compute" $V$ in such a case? What that would even mean? Each eigenvector has infinitely many coordinates! $\endgroup$ – amoeba Jan 2 '15 at 22:36
  • $\begingroup$ @amoeba, thanks for the comment. Please consider my revision. $\endgroup$ – Simon Kuang Jan 2 '15 at 23:46
  • $\begingroup$ Hmm, remember that all eigenvectors have (by construction) the length $1$. So what you want is definitely not "the length of the eigenvector". I am not sure I understand your actual goal; maybe it is going to be helpful if you spend more time explaining the question that you actually want to get an answer to. Then we can try to formalize it mathematically. $\endgroup$ – amoeba Jan 3 '15 at 0:04
  • $\begingroup$ @amoeba Sorry about the confusion. I meant the principal components, these vectors. $\endgroup$ – Simon Kuang Jan 3 '15 at 0:05
  • $\begingroup$ But these are just [square roots of] the respective eigenvalues, no? You directly compute them in kernel PCA. They seem to have nothing to do with "integrating length for the input-space feature PC projections" (as much as I can understand what this should mean). $\endgroup$ – amoeba Jan 3 '15 at 0:07
3
$\begingroup$

[I apologize that I have not posted this answer earlier, as promised.]

This question seems to have been built upon series of confusions (partly clarified in the comments above and reflected in the edits).

Q1: How can one obtain eigenvectors [of the covariance matrix] in the target space?

This is generally not possible. For some kernels, like e.g. Gaussian kernel, the target space is infinite-dimensional. There is no way one can obtain an eigenvector there: it would need to have infinitely many coordinates! The beauty of the "kernel trick" is that it allows to compute projections of the data onto this eigenvector without ever computing the eigenvector itself.

Q2: Okay, but can one at least compute the length of an eigenvector?

Yes. The length is equal to $1$, because that's how we choose to normalize eigenvectors.

Q3: But what about the length of an eigenvector, like it's depicted on 2D examples of PCA?

This length is simply the square root of the respective eigenvalue. They are directly computed in kPCA.


Having said all that, here is my best guess about what you actually wanted to get.

I want to be able to have, for a "classification" PCA case as this one, a graph like the one on the left showing a curved (for lack of a better word) eigenvector projected down to the input space. I'd prefer a parametric form over which I may compute an arclength integral.

It seems you want to get the length of the decision boundary between two classes in the original feature space. The main problem here is that kPCA is not a classification technique. There is no way kPCA will give you a decision boundary: it's simply projects all your data to some subspace of the target space. If you are lucky, the classes will be linearly separated. But if you need a decision boundary, you will need to use some other, additional, approach to obtain it, and it should be a supervised algorithm (kPCA is unsupervised).

After you come up with or choose an algorithm, then you can ask yourself how to compute the length of the decision boundary. E.g. perhaps you can use some linear classifier (such as LDA, logistic regression, or linear SVM) to get a linear boundary in the kPCA space, and then ask what is the length of the corresponding boundary in the original space. It is not going to be very easy, as far as I can see.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.