The PCA optimization problem is known as $$ \max_{U \in \mathbb{R}^{d\times r}, U^TU = I} tr(U^T\Sigma U), $$ where $\Sigma$ is a covariance matrix of a dataset $\{x_1,\dots,x_n\} \subset \mathbb{R}^d$, that we suppose with zero mean, so $$\Sigma = \sum_{i=1}^n xx^T. $$

Kernel PCA consists in applying a map $\phi \colon \mathbb{R}^d \to \mathcal{F}$ to the dataset, where $\mathcal{F}$ is some Hilbert space, and learning PCA in this new space with the transformed data. The key item of this problem is that we just need to know the dot products in $\mathcal{F}$ to be able to optimize in the new space.

One of the advantages of needing only the dot products is that $\mathcal{F}$ can be even infinite dimensional. But in this case, how should I understand the PCA problem in this space? In this case we can't talk about matrices. So I suppose that the matrix $U$ is replaced by a bounded linear operator $U \colon \mathcal{F} \to \mathbb{R}^r$. And maybe the condition $U^TU = I$ could be replaced by "$U$ is a partial isometry on a subspace of $\mathcal{F}$". Is this right? And how should I understand the covariance $\Sigma$, and the expression $U^T\Sigma U$? And what about the trace?

Finally, as a bonus question, how can I establish rigurously a representer theorem for kernel PCA when transformed inputs are infinite dimensional? A representer theorem is a theorem that states that the solution vectors that compose the matrix $U$ are in the span of the transformed data $\{\phi(x_1),\dots,\phi(x_n)\}$. This theorem is needed to be able to express the solution in terms of a finite matrix.

  • $\begingroup$ I could try to answer this. Are you still around, @utbutnut? $\endgroup$
    – amoeba
    Sep 16, 2018 at 16:10
  • $\begingroup$ I have some more intuition, but I'm still interested in a rigurous explanation. $\endgroup$
    – utbutnut
    Sep 18, 2018 at 15:48


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.