1
$\begingroup$

I'm performing a sort of weighted kernel PCA, where the weights of samples can be negative. The weights of all samples are given by the diagonal weight matrix $D$. The data matrix is the $n \times d$ matrix $X$.

This way I obtain the 'principal components' is by computing the eigenvectors of $X^T D X$. Thus the 'principal component' $u$ is given by:

$M u = X^T D X u = \lambda u$ (1)

Where $M = X^T D X$. (I use no centering of the datamatrix).

Now what I want to know is the following. Let's say I train a model $w$. Now I want to compute $w^T u / ||u||$, the projection of $w$ on $u$. Only I want to do this in a particular kernel $K$, the same kernel which I use for this kind of weighted PCA.

I think, we can derrive the innerproduct as follows, but I'm not sure if this is entirely correct. I follow the derrivation of regular kernel PCA given here. I assume that it still holds that $u$ can be written as a linear combination of the data (as in regular PCA): $u = X^T \alpha$. Substituting this in (1) we find that:

$$X^T D X X^T \alpha = \lambda X^T \alpha$$

Now left multiplying by $D X X^T$ we have that:

$$D X X^T D X X^T \alpha = \lambda D X X^T \alpha$$

Defining the kernel matrix by $K = XX^T$ we find:

$$D K D K \alpha = \lambda D K \alpha$$

The matrix $KD$ is however not symmetric, which is troubling. So it is questionable if we can obtain $\alpha$ using the eigenvectors of $KD$ as in 1. Assuming for now that we can then we can find $\alpha$ by solving: $$D K \alpha = \lambda \alpha$$ Then to obtain $w^T u$ we compute (note that $w = X^T c$ due to the representer theorem, and $u = X^T \alpha$):

$$w^T u = c^T X X^T \alpha = c^T K \alpha$$

And $||u||^2 = u^T u = \alpha^T K \alpha$. Then thus we have that:

$$w^T u / |u| = c^T K \alpha / \alpha^T K \alpha$$

Only as mentioned, $KD$ is not symmetric! So how to find the vector $\alpha$...?

Some related results: $X^T D X$ and $D X X^T$ have the same eigenvalues since $AB$ and $BA$ always have the same eigenvalues. If $\alpha$ is an eigenvector of $X^T X D$, and $u$ is an eigenvector of $X^T D X$, then we can show that $\alpha = X u$ (which seems to be just the other way around of what I need...!), see my question here. Any help would be greatly appreciated!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.