3
$\begingroup$

Could someone please help me understand why when we project a new sample in kernel PCA to an eigenvector $\alpha$, we normalize it by dividing the eigenvector $\alpha$ with its eigenvalue $\lambda$?

I understand that we need to normalize the eigenvector $V$ and hence the eigenvector $\alpha$ following the criteria described in Eq 9 of the following original paper: Scholkopf et al. 1997, Kernel Principal Component Analysis.

However, I couldn't figure out why we divide by $\lambda$ in the normalization (instead of e.g. $(\|\boldsymbol\alpha^{k}\|\sqrt\lambda)$ - see below for derivation)?

Eq.9 requires

$$1 = \lambda_k(\boldsymbol\alpha^{k*}\cdot\boldsymbol\alpha^{k*})$$

where $\boldsymbol\alpha^{k*}$ is the normalized $\boldsymbol\alpha^k$. Since $\boldsymbol\alpha^k\cdot\boldsymbol\alpha^k = \|\boldsymbol\alpha^k\|^2$,

$$\boldsymbol\alpha^{k*} = \frac{\boldsymbol\alpha^{k}}{\|\boldsymbol\alpha^{k}\|\sqrt{\lambda_k}}$$

Eq. 10 (projection of a test point $\boldsymbol\phi(\mathbf{x})$) becomes

$$(\mathbf{V}^{k*}\cdot\boldsymbol\phi(\mathbf{x})) = \sum_{i=1}^{l}\alpha_i^{k*} (\boldsymbol\phi(\mathbf{x}_i)\cdot\boldsymbol\phi(\mathbf{x})) = \frac{1}{\|\boldsymbol\alpha^{k}\|\sqrt{\lambda_k}}\sum_{i=1}^{l} \alpha_i^{k}(\boldsymbol\phi(\mathbf{x}_i)\cdot\boldsymbol\phi(\mathbf{x}))$$ $$= \frac{1}{\sqrt{\lambda_k}}\sum_{i=1}^{l} \alpha_i^{k}(\boldsymbol\phi(\mathbf{x}_i)\cdot\boldsymbol\phi(\mathbf{x}))$$

where $\mathbf{V}^{k*}$ is the normalized $\mathbf{V}^{k}$ and $\|\boldsymbol\alpha^k\| = 1$. But, I know that the above normalization is incorrect because instead of dividing by $\sqrt{\lambda_k}$, we should simply divide by $\lambda_k$. So, I am not sure where I am making mistakes here.

For a nice python implementation of kernel PCA and the normalization of the eigenvector, please refer to the following site: http://sebastianraschka.com/Articles/2014_kernel_pca.html You can go straight to

def project_x(x_new, X, gamma, alphas, lambdas):

to see that we need to normalize the $\alpha$ by dividing it with $\lambda$ in order to get the correct projection.

Update:

The above normalization is correct. The question was there because I misunderstood Sebastian's code.

$\endgroup$
23
  • 2
    $\begingroup$ It seems to me that in order to get $\alpha^k \cdot \alpha^k = 1$ (normalization), you need to divide by $\lambda$. This is simply algebra as per Eqn 9. Or are you asking to clarify what's happening in Eqn 9? $\endgroup$
    – ilanman
    Commented Dec 22, 2016 at 23:15
  • 2
    $\begingroup$ I think I can see what happens in that code. He computes X_pc without any normalization (whereas "correct" normalization involves a square root of lambda - see my linked answer). So then an "out of sample" projection he computes with 1/sqrt(lambda) instead of 1/lambda and it fits to the PCs because those were NOT multiplied by sqrt(lambda). $\endgroup$
    – amoeba
    Commented Dec 25, 2016 at 22:01
  • 5
    $\begingroup$ @amoeba: i am reading your derivation again, and will just post the question on your answer in the link: stats.stackexchange.com/questions/126014/…. Thank you for helping.. really appreciate it.. $\endgroup$
    – Starz
    Commented Dec 26, 2016 at 2:41
  • 1
    $\begingroup$ I edited my linked answer to clarify things, see if it became clearer. (1) What I call "Gram matrix" for data matrix X is is XX'. Covariance matrix is X'X/n. See my edits. (2) X is a data matrix not a covariance matrix. Again, see my edits. $\endgroup$
    – amoeba
    Commented Dec 26, 2016 at 10:22
  • 1
    $\begingroup$ See my answer in stats.stackexchange.com/questions/134282 about SVD/PCA connection. I will take a look at your other question later. $\endgroup$
    – amoeba
    Commented Dec 26, 2016 at 17:01

0