2
$\begingroup$

Given a data matrix $\mathbf X$, can I always obtain its covariance matrix (to use in PCA) by centering (subtracting the column means) and then computing $\mathbf X \mathbf X^\top/n$? Is this always valid? Can I always center the data by subtracting the mean? Are there any restrictions when this cannot be applied?


I am working on quantum algorithms for principal component analysis (PCA). There exists a quantum algorithm which can calculate the PCA exponentially faster (at least in theory), but it works only if the covariance matrix is given in the Gram form, i.e. in the $\mathbf X\mathbf X^\top$ form. That is why I am wondering if there are any restrictions for this form?

The relationship between SVD and PCA is explained in this excellent post but it couldn't answer my question.

$\endgroup$
  • 2
    $\begingroup$ First of all, how do you define "Gram matix"? This very ambiguous, misused term sometimes mean here or there $X'X$ and sometimes $XX'$ and sometimes Gramian matrix in the sense "positive semidefinite" or some other sense... Second, in your first link I could not find any mentioning "Gram". Third, this question addresses the issue of precision in the dilemma "svd or eigen". $\endgroup$ – ttnphns May 6 '16 at 8:43
  • 3
    $\begingroup$ Forth, in a broad sense, PCA - does not require centering; centering is seen as a pre-processing step. But results and their interpretation depend whether you center or not, of course. $\endgroup$ – ttnphns May 6 '16 at 8:44
  • 2
    $\begingroup$ @ttnphns The principal components are defined as the eigenvectors of the covariance matrix. Subtracting the expected value is not optional when speaking of covariance. $\endgroup$ – broncoAbierto May 6 '16 at 10:46
  • 1
    $\begingroup$ No restrictions, you can always subtract the mean to get the "centered" data and then $XX^\top/n$ will equal the covariance matrix. $\endgroup$ – amoeba May 6 '16 at 22:16
  • 2
    $\begingroup$ @Cagdas You mean that sample covariance matrix will be a bad estimator of the population covariance matrix? Can be, yes. But sample covariance matrix is just XX'/n by definition. $\endgroup$ – amoeba May 11 '16 at 16:08
2
$\begingroup$

This is the difference between centered (or demeaned) and uncentered PCA. One recent (2016) paper that among others discuss this is Principal component analysis:a review and recent developments by Ian T. Jolliffe and Jorge Cadima, which can be found here. I will just cite what they say:

As was seen in §2, PCA amounts to an SVD of a column-centred data matrix. In some applications [see below], centring the columns of the data matrix may be considered inappropriate. In such situations, it may be preferred to avoid any pre-processing of the data and to subject the uncentred data matrix to an SVD or, equivalently, to carry out the eigendecomposition of the matrix of noncentred second moments, T, whose eigenvectors define linear combinations of the uncentred variables. This is often referred to as an uncentred PCA and there has been an unfortunate tendency in some fields to equate the name SVD only with this uncentred version of PCA. Uncentred PCs are linear combinations of the uncentred variables which successively maximize non-central second moments, subject to having their crossed non-central second moments equal to zero. Except when the vector of column means x¯ (i.e. the centre of gravity of the original n-point scatterplot in p-dimensional space) is near zero (in which case centred and uncentred moments are similar), it is not immediately intuitive that there should be similarities between both variants of PCA. Cadima & Jolliffe ON RELATIONSHIPS BETWEEN UNCENTRED AND COLUMN-CENTRED PRINCIPAL COMPONENT ANALYSIS have explored the relations between the standard (column-centred) PCA and uncentred PCA and found them to be closer than might be expected, in particular when the size of vector $\bar{x}$ is large. It is often the case that there are great similarities between many eigenvectors and (absolute) eigenvalues of the covariance matrix S and the corresponding matrix of non-centred second moments, T. In some applications, row centrings, or both row- and column-centring (known as doublecentring) of the data matrix, have been considered appropriate. The SVDs of such matrices give rise to row-centred and doubly centred PCA, respectively.

One case when centering often is unnatural and doesn't help is for images. One example on this site is What is the intuition behind SVD?.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.