PCA, Covariance of predictors and Noise/Distortion of Data

Question

I am reading a note on PCA and in the introduction the author states that large diagonal values in the covariance matrix of the predictors correspond to a strong signal and large off-diagonal values --i.e. covariances-- correspond to high noise or distortion in our data.

He furthermore presents an example from the well-known iris dataset which can be found in base R among other places.

> cov(iris[1:4])
             Sepal.Length Sepal.Width Petal.Length Petal.Width
Sepal.Length    0.6856935  -0.0424340    1.2743154   0.5162707
Sepal.Width    -0.0424340   0.1899794   -0.3296564  -0.1216394
Petal.Length    1.2743154  -0.3296564    3.1162779   1.2956094
Petal.Width     0.5162707  -0.1216394    1.2956094   0.5810063

In this dataset the covariances between Petal.Length and Sepal.Length and Petal.Width and Petal.Length are rather high.

I can intuitively understand the statement that large variances in the predictors imply a strong signal. I can also readily comprehend that large covariances signify redundancy in the Signal, i.e. high multicollinearity.

What I find difficult to understand intuitively is that high covariances relate to high noise and distortion in the data. For a starter, in the particular example of iris data brought forward by the author, the fact that certain attributes of the flowers appear correlated is in no way a distortion or a noise, simply a physical reality. But perhaps I do not comprehend the notion of distortion and noise as understood by the author of the note.

Your advice will be appreciated.

Answer 1

My understanding is that PCA does not have a concept of "noise", as opposed to other similar methods like factor analysis or probabilistic PCA that come with a model for the data in terms of a signal + noise decomposition. PCA simply determines how well the data can be represented by projections into lower dimension subspaces, and the off-diagonal elements, as you rightly say, tell us about the (pairwise) multicollinearity. It certainly works better on data coming from a Gaussian distribution than data coming from more irregular distributions, but this isn't a requirement.

I wonder what the author would say to the case where we have $n$ observations on one variable $x := (x_1, \dots, x_n)^T$ and our data matrix is $X = (x \hspace{3mm} x) \in \mathbb R^{n \times 2}$, so that $X^T X = x^T x \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$. This ought to be the perfect setting for PCA since we can losslessly project $X$ into a 1-dimensional subspace. If anything, this is a much better setting than when $X^T X$ is diagonal because then we don't gain anything, and PCA just corresponds to deleting columns of $X$ in order of increasing variance.

Could it be that there is some extra context that we are missing from this question, such as how each variable is supposed to be independent so $E(X^T X)$ is diagonal?