# PCA vs Least Squares

Principal Component Analysis By SVANTE WOLD, Chemometrics and Intelligent Laboratory Systems, 2 (1987) 37-52

In the above paper, the author says... "As PCA is a least squares method, outlier severely influence the model"

By what I have understood, PCA is transformation technique where orthogonal transformation are done to better explain the data. Why does author say PCA is a least squares method?

Given $$m$$ vectors $$\boldsymbol{x}_1, \ldots, \boldsymbol{x}_m \in \mathbb{R}^n$$, find matrices $$\boldsymbol{U} \in \mathcal{M}_{\mathbb{R}}(k, n)$$ and $$\boldsymbol{V} \in \mathcal{M}_{\mathbb{R}}(n, k)$$ such that $$\sum_{i=1}^m{||\boldsymbol{x}_i - \boldsymbol{V}\boldsymbol{U}\boldsymbol{x}_i ||}^2$$ is minimized.
That is, for $$k < n$$ the vector $$\boldsymbol{U}\boldsymbol{x}_i \in \mathbb{R}^k$$ is the projection of $$\boldsymbol{x}_i$$ into a lower-dimensional subspace, and $$\boldsymbol{V}\boldsymbol{U}\boldsymbol{x}_i$$ is the reconstructed original vector. PCA aims to find matrices $$\boldsymbol{U}, \boldsymbol{V}$$ that minimize the reconstruction error as measured by the $$\ell^2$$-norm. It can be shown that, in fact, these matrices are orthogonal and $$\boldsymbol{U} = \boldsymbol{V}^T$$, so the problem reduces to $$\underset{V \in \mathcal{M}_{\mathbb{R}}(n,k)}{\mathrm{arg\,min}}\sum_{i=1}^m{||\boldsymbol{x}_i - \boldsymbol{V}\boldsymbol{V}^T\boldsymbol{x}_i ||}^2\,.$$ Further manipulations show that $$\boldsymbol{V}$$ is the matrix whose columns are the eigenvectors corresponding to the $$k$$ largest eigenvalues of $$\sum_{i=1}^m \boldsymbol{x}_i{\boldsymbol{x}_i}^T\,,$$ as expected. So indeed, PCA is a least squares method and it is quite sensitive to outliers.