3
$\begingroup$

Say $\mathbf{X} \in \mathbb{R}^{n \times p}$ and $\boldsymbol{\Sigma} = \frac{1}{n}\mathbf{X}'\mathbf{X}$. The eigenvector decomposition of $\boldsymbol{\Sigma}$ gives $\boldsymbol{\Sigma} = \mathbf{P}\boldsymbol{\Lambda}\mathbf{P}'$ where $\mathbf{P}$ are the eigenvectors and $\boldsymbol{\Lambda}$ is a diagonal matrix with eigenvalues on the diagonal. The singular value decomposition of $\mathbf{X}$ gives $\mathbf{X}=\mathbf{U}\mathbf{S}\mathbf{V}'$, where $\mathbf{U}$ contains the left eigenvectors of $\mathbf{X}$, $\mathbf{V}$ contains the right eigenvectors of $\mathbf{X}$, and $\mathbf{S}$ is a diagonal matrix with singular values on the diagonal. The eigenvalues of $\boldsymbol{\Sigma}$ will equal the squares of the singular values of $\mathbf{X}$ only if $\mathbf{X}$ was centered (the mean of $\mathbf{X}$ was removed). If I plot the cumulative variance explained by the principal components, the plot will (obviously) be different depending on whether $\mathbf{X}$ was centered first.

I'm trying to understand what the eigenvectors mean if I don't center $\mathbf{X}$ prior to doing the decomposition. I'm working on an naive algorithm that involves performingPCA online. I need to recompute eigenvectors after observing each new data point, and my results are much worse if I recenter the data set as I observe new data points. However, if I pretend to have seen all the data at first, center the data, and compute the eigenvectors of the entire centered data set, my (offline) performance is much better than if I had not centered my data.

Could someone please provide some intuition as to what centering the data set does and how this might be affecting my algorithm performance? Also, what is the difference in the singular values computed before and after centering the data?

Improve this question
$\endgroup$
5
  • 3
    $\begingroup$ Svd and eigen-decomposition are always closely tied. Regardless of whether $\bf X$ had been centered/scaled or not, svd is $\bf X=USV'$ and eigen is $\bf X'X=VS^2V'$ or $\bf XX'=US^2U'$. Of course, centering prior the decomposition yield different eigenvectors and eigenvalues than without centering, because principal axes will come through the origin in any case, but the data ellipsoid changes its position relative the origin. $\endgroup$
    – ttnphns
    Commented Nov 29, 2014 at 17:33
  • $\begingroup$ Why do you say you need to redo eigen after each new data point entered? $\endgroup$
    – ttnphns
    Commented Nov 29, 2014 at 17:42
  • 3
    $\begingroup$ Covariance matrix always implies centering (stats.stackexchange.com/a/22520/3277). Without centering, it is just the (divided by n or n-1) SSCP matrix. $\endgroup$
    – ttnphns
    Commented Nov 29, 2014 at 17:47
  • $\begingroup$ First I think it is then using the data as on ratio-scale. Second I think it can be seen by the following: assume a cloud of data-points in a 2-dim scatterplot in the first quadrant, and the main axis of the ellipse has negative slope. Then centering means: we shift the cloud to the origin and find then the principal axes: the result is that the first principal axe has the negative slope and the second is orthogonal to this. If data are not centeres, the vector of main direction from the origin through the data has positive slope, and the second is roughly that of the first previously $\endgroup$
    – Gottfried Helms
    Commented Nov 30, 2014 at 9:54
  • $\begingroup$ Hello @vman049. I don't think I understand the performance you describe (btw, what performance? are you doing classification/prediction?). If it is advantageous to centre the data offline, then it should also be advantageous to do it online. Presumably, as you observed almost all data points in the online algorithm, it should perform similarly to the offline one. How can it possibly perform worse? Could you please clarify? $\endgroup$
    – amoeba
    Commented Dec 5, 2014 at 13:06

0

Reset to default

Browse other questions tagged or ask your own question.