I'm performing a sort of weighted kernel PCA, where the weights of samples can be negative. The weights of all samples are given by the diagonal weight matrix $D$. The data matrix is the $n \times d$ matrix $X$.

This way I obtain the 'principal components' is by computing the eigenvectors of $X^T D X$. Thus the 'principal component' $u$ is given by:

$M u = X^T D X u = \lambda u$ (1)

Where $M = X^T D X$. (I use no centering of the datamatrix).

Now what I want to know is the following. Let's say I train a model $w$. Now I want to compute $w^T u / ||u||$, the projection of $w$ on $u$. Only I want to do this in a particular kernel $K$, the same kernel which I use for this kind of weighted PCA.

I think, we can derrive the innerproduct as follows, but I'm not sure if this is entirely correct. I follow the derrivation of regular kernel PCA given here. I assume that it still holds that $u$ can be written as a linear combination of the data (as in regular PCA): $u = X^T \alpha$. Substituting this in (1) we find that:

$$X^T D X X^T \alpha = \lambda X^T \alpha$$

Now left multiplying by $D X X^T$ we have that:

$$D X X^T D X X^T \alpha = \lambda D X X^T \alpha$$

Defining the kernel matrix by $K = XX^T$ we find:

$$D K D K \alpha = \lambda D K \alpha$$

The matrix $KD$ is however not symmetric, which is troubling. So it is questionable if we can obtain $\alpha$ using the eigenvectors of $KD$ as in 1. Assuming for now that we can then we can find $\alpha$ by solving: $$D K \alpha = \lambda \alpha$$ Then to obtain $w^T u$ we compute (note that $w = X^T c$ due to the representer theorem, and $u = X^T \alpha$):

$$w^T u = c^T X X^T \alpha = c^T K \alpha$$

And $||u||^2 = u^T u = \alpha^T K \alpha$. Then thus we have that:

$$w^T u / |u| = c^T K \alpha / \alpha^T K \alpha$$

Only as mentioned, $KD$ is not symmetric! So how to find the vector $\alpha$...?

Some related results: $X^T D X$ and $D X X^T$ have the same eigenvalues since $AB$ and $BA$ always have the same eigenvalues. If $\alpha$ is an eigenvector of $X^T X D$, and $u$ is an eigenvector of $X^T D X$, then we can show that $\alpha = X u$ (which seems to be just the other way around of what I need...!), see my question here. Any help would be greatly appreciated!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.