This question already has an answer here:

I wanted to conduct a total least squares regression on two variables. My statistical programme does not provide TLS, but TLS luckily equals Principal Component Analysis, as far as I know. Since all variables are standardized, this is done by applying a SVD on the correlation matrix of the concerned variates.

Now, my take on the issue was to read out the first eigenvector to obtain the TLS coefficients. However, the SVD gives me the same eigenvector (weights) irrespective of what the two variables are. It's always [.70710678, .70710678]. I find this strange. Of course, the eigenvalues differ.

My questions are: How to interpret this? Has this result maybe anything to do with the coordinate space, like it is just a matter of rotation? One note: I used Stata to apply the SVD. To be clear: The question is not about TLS directly, but why I get the same eigenvectors irrespective of which variables I use (as long as they are exactly 2).


marked as duplicate by kjetil b halvorsen, whuber Feb 19 at 13:58

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ Very related: stats.stackexchange.com/questions/19300 (basically duplicate). $\endgroup$ – amoeba Mar 30 '17 at 13:13
  • $\begingroup$ @whuber Just saw that this was closed as a duplicate. I find that this thread could be a better duplication target than the other one. It has many more upvotes and views, despite being much more recent. $\endgroup$ – amoeba Feb 19 at 20:52

Algebraically, correlation matrix for two variables looks like that: $$\begin{pmatrix} 1 & \rho \\ \rho & 1 \end{pmatrix}.$$ Following the definition of an eigenvector, it is easy to verify that $(1, 1)$ and $(-1, 1)$ are the eigenvectors irrespective of $\rho$, with eigenvalues $1+\rho$ and $1-\rho$. For example:

$$\begin{pmatrix} 1 & \rho \\ \rho & 1 \end{pmatrix}\begin{pmatrix}1\\1\end{pmatrix}=(\rho+1)\begin{pmatrix}1\\1\end{pmatrix}.$$

Normalizing these two eigenvectors to unit length yields $(\sqrt{2}/2, \sqrt{2}/2)$ and $(-\sqrt{2}/2, \sqrt{2}/2)$, as you observed.

Geometrically, if the variables are standardized, then the scatter plot will always be stretched along the main diagonal (which will be the 1st PC) if $\rho>0$, whatever the value of $\rho$ is:

Two standardized variables with various correlation coefficients

Regarding TLS, you might want to check my answer in this thread: How to perform orthogonal regression (total least squares) via PCA? As should be pretty obvious from the figure above, if both your $x$ and $y$ are standardized, then the TLS line is always a diagonal. So it hardly makes sense to perform TLS at all! However, if the variables are not standardized, then you should be doing PCA on their covariance matrix (not on their correlation matrix), and the regression line can have any slope.

  • 2
    $\begingroup$ I get different eigenvalues when solving the characteristics equation manually: (1-λ)^2 - 𝜌^2 = (1-λ-𝜌)(1-λ+𝜌). λ(1) = 1-𝜌; λ(2) = 1+𝜌; $\endgroup$ – Sergei Rodionov Feb 19 at 8:46
  • $\begingroup$ You are right! Fixed. $\endgroup$ – amoeba Feb 19 at 8:50

As your first eigenvector is $(\sqrt{2}, \sqrt{2})$, the other eigenvector is uniquely (we're in 2D) up to factor $1$/$-1$ the vector $(\sqrt{2} -\sqrt{2})$. So you get your diagonalizing orthogonal matrix as $$\sqrt{2}\left[ \begin{array}{cc} 1 & 1 \\ 1 & -1 \end{array} \right]$$

No we can reconstruct the covariance* matrix to have the shape $$\left[ \begin{array}{cc} a+b & a-b \\ a-b & a+b \end{array} \right] $$ $a$ and $b$ are the eigenvalues. I would suggest to look closely on your model or the origin of the data. Then you might find a reason why your data may be distributed as $X_1=X_a + X_b$ and $X_2 = X_a - X_b$, where $Var(X_a)=a$ and $Var(X_b)=b$ and $X_a$ and $X_b$ are independent.

If your data would follow a continuous multivariate distribution, it is almost sure that your correlation matrix follows from this sum/difference relation. If the data follow a discrete distribution, it is still very likely that the model $X_1=X_a + X_b$ and $X_2 = X_a - X_b$ describes your data properly. In this case, you don't need a PCA.

But it is generally better to infer such relations from sure insight into the nature of the data and not by estimation procedures like PCA.

*Say correlation matrix, if $a+b=1$.

  • 1
    $\begingroup$ I am confused by this answer. Why is it "obvious"? Also, the claim in the question is that the eigenvectors of any 2x2 correlation matrix are the same. Is it true? If so, why? The model or origin of the data are of no relevance here. $\endgroup$ – amoeba Mar 5 '15 at 11:27
  • $\begingroup$ The model and origin of the data are of relevance, because that's what the question is about. I'll clarify the word "obvious", because you're right, this word is always a trap for proofs. $\endgroup$ – Horst Grünbusch Mar 5 '15 at 11:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.