PCA vs Least Squares Principal Component Analysis By SVANTE WOLD, Chemometrics and Intelligent Laboratory Systems, 2 (1987) 37-52
In the above paper, the author says... "As PCA is a least squares method, outlier severely influence the model"
By what I have understood, PCA is transformation technique where orthogonal transformation are done to better explain the data. Why does author say PCA is a least squares method?
 A: PCA can be formulated as follows:

Given $m$ vectors $\boldsymbol{x}_1, \ldots, \boldsymbol{x}_m \in \mathbb{R}^n$, find matrices $\boldsymbol{U} \in \mathcal{M}_{\mathbb{R}}(k, n)$ and $\boldsymbol{V} \in \mathcal{M}_{\mathbb{R}}(n, k)$ such that $$\sum_{i=1}^m{||\boldsymbol{x}_i - \boldsymbol{V}\boldsymbol{U}\boldsymbol{x}_i ||}^2$$ is minimized.

That is, for $k < n$ the vector $\boldsymbol{U}\boldsymbol{x}_i \in \mathbb{R}^k$ is the projection of $\boldsymbol{x}_i$ into a lower-dimensional subspace, and $\boldsymbol{V}\boldsymbol{U}\boldsymbol{x}_i$ is the reconstructed original vector. PCA aims to find matrices $\boldsymbol{U}, \boldsymbol{V}$ that minimize the reconstruction error as measured by the $\ell^2$-norm. It can be shown that, in fact, these matrices are orthogonal and $\boldsymbol{U} = \boldsymbol{V}^T$, so the problem reduces to
$$\underset{V \in \mathcal{M}_{\mathbb{R}}(n,k)}{\mathrm{arg\,min}}\sum_{i=1}^m{||\boldsymbol{x}_i - \boldsymbol{V}\boldsymbol{V}^T\boldsymbol{x}_i ||}^2\,.$$
Further manipulations show that $\boldsymbol{V}$ is the matrix whose columns are the eigenvectors corresponding to the $k$ largest eigenvalues of
$$\sum_{i=1}^m \boldsymbol{x}_i{\boldsymbol{x}_i}^T\,,$$
as expected. So indeed, PCA is a least squares method and it is quite sensitive to outliers.
