Principal Component Analysis By SVANTE WOLD, Chemometrics and Intelligent Laboratory Systems, 2 (1987) 37-52

In the above paper, the author says... "As PCA is a least squares method, outlier severely influence the model"

By what I have understood, PCA is transformation technique where orthogonal transformation are done to better explain the data. Why does author say PCA is a least squares method?


PCA can be formulated as follows:

Given $m$ vectors $\boldsymbol{x}_1, \ldots, \boldsymbol{x}_m \in \mathbb{R}^n$, find matrices $\boldsymbol{U} \in \mathcal{M}_{\mathbb{R}}(k, n)$ and $\boldsymbol{V} \in \mathcal{M}_{\mathbb{R}}(n, k)$ such that $$\sum_{i=1}^m{||\boldsymbol{x}_i - \boldsymbol{V}\boldsymbol{U}\boldsymbol{x}_i ||}^2$$ is minimized.

That is, for $k < n$ the vector $\boldsymbol{U}\boldsymbol{x}_i \in \mathbb{R}^k$ is the projection of $\boldsymbol{x}_i$ into a lower-dimensional subspace, and $\boldsymbol{V}\boldsymbol{U}\boldsymbol{x}_i$ is the reconstructed original vector. PCA aims to find matrices $\boldsymbol{U}, \boldsymbol{V}$ that minimize the reconstruction error as measured by the $\ell^2$-norm. It can be shown that, in fact, these matrices are orthogonal and $\boldsymbol{U} = \boldsymbol{V}^T$, so the problem reduces to $$\underset{V \in \mathcal{M}_{\mathbb{R}}(n,k)}{\mathrm{arg\,min}}\sum_{i=1}^m{||\boldsymbol{x}_i - \boldsymbol{V}\boldsymbol{V}^T\boldsymbol{x}_i ||}^2\,.$$ Further manipulations show that $\boldsymbol{V}$ is the matrix whose columns are the eigenvectors corresponding to the $k$ largest eigenvalues of $$\sum_{i=1}^m \boldsymbol{x}_i{\boldsymbol{x}_i}^T\,,$$ as expected. So indeed, PCA is a least squares method and it is quite sensitive to outliers.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.