11
$\begingroup$

If we have some centered dataset $X$ then the eigenvectors of $X^TX$ represent the principal components of the dataset, and their physical meaning is the directions that data follow in the original feature space.

Now, in the same way as before, we can create the Kernel matrix $XX^T$ of the centered data and we can compute its eigenvectors and eigenvalues.

I understand that there might be some benefits in utilising this decomposition, like utilising a subset of the top components as discussed for example in this paper by J. Shawe-Taylor et al., however, I am not sure if I understand the physical meaning of this decomposition. What are the directions now, and what's the meaning of the eigenvalues?

Improve this question
$\endgroup$
2
  • $\begingroup$ Is centering in both cases the same - the centering of every column of X? $\endgroup$
    – ttnphns
    Commented Mar 17, 2016 at 5:19
  • $\begingroup$ Yes, I refer to centering every column. $\endgroup$
    – iliasfl
    Commented Mar 18, 2016 at 0:50

1 Answer 1

Reset to default
12
$\begingroup$

The eigenvalues are actually the same as those of the covariance matrix. Let $X = U \Sigma V^T$ be the singular value decomposition; then $$X X^T = U \Sigma \underbrace{V^T V}_{I} \Sigma U^T = U \Sigma^2 U^T$$ and similarly $X^T X = V \Sigma^2 V^T$. Note that in the typical case where $X$ is $n \times p$ with $n \gg p$, most of the eigenvalues of the Gram matrix will be zero. If you're using say an RBF kernel, none will be zero (though some will probably be incredibly small).

The eigenvectors of the Gram matrix are thus seen to be the left singular values of $X$, $U$. One way to interpret these is:

  • The right singular vectors (columns of $V$, the eigenvectors of the covariance matrix) give the directions that data tends to lie on in the feature space.
  • The singular values (diagonal of $\Sigma$, square root of the eigenvalues of either matrix) give how important each component is to the dataset as a whole.
  • The left singular vectors (columns of $U$, the eigenvectors of the Gram matrix) give the representation of how much each data point is represented by each of the components, relative to how much they're used in the whole dataset. (Columns of $U \Sigma$ give the scores, the linear coefficient of each component when representing the data in the basis $V$.)

If you take only the first few columns of $U$ (and the corresponding block of $\Sigma$), you get the data projected as well as possible onto the most frequent components (PCA).

If a data point has high norm of its row of $U$, that means that it uses components much more than other components do, ie it has high leverage / "sticks out." If $p > n$, these will all be one, in which case you can either look only at the first $k$ values (leverage scores corresponding to the best rank-k approximation) or do some kind of soft-thresholding instead. Doing that with $k=1$, which is computationally easier, gives you PageRank.

See also this thread and the links therein for everything you'd ever want to know about SVD/PCA, which you perhaps didn't realize was really your question but it was.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ +1. Another related thread: Is PCA still done via the eigendecomposition of the covariance matrix when dimensionality is larger than the number of observations? $\endgroup$
    – amoeba
    Commented Mar 17, 2016 at 10:17
  • $\begingroup$ Small typos above: In general, $\Sigma$ is (say) an $n\times p$ matrix; so $\Sigma \ne \Sigma^\top$. In particular, we have $X X^\top = U \Sigma \Sigma^\top U$ and $X^\top X = V \Sigma^\top \Sigma V$. Assume $n\ge p$. Since $\Sigma$ is a rectangular diagonal matrix, the $p\times p$ upper left submatrix of $\Sigma \Sigma^\top$ is exactly $\Sigma^\top \Sigma$. The other case is symmetric. $\endgroup$
    – Csaba
    Commented Oct 30, 2023 at 20:40
  • $\begingroup$ @Csaba I should have clarified, but I usually prefer the “thin” SVD where $\Sigma$ is square; I think everything above is fine with that version of it. $\endgroup$
    – Danica
    Commented Oct 31, 2023 at 2:32
  • 1
    $\begingroup$ I see. So, when $n\ge p$, $U$ is $n\times p$, $\Sigma$ is $p\times p$. It makes sense! Thanks for the clarification. $\endgroup$
    – Csaba
    Commented Oct 31, 2023 at 4:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.