11
$\begingroup$

Suppose $\Sigma$ is a covariance matrix $P$ is its corresponding correlation matrix. Let $\lambda_1, \dots, \lambda_p$ and $\tau_1, \dots, \tau_p$ denote the ordered eigenvalues of $\Sigma$ and $P$, respectively.

  • Is there any known relationship between $\lambda_i$ and $\tau_i$?
  • Is there any known relationship between their corresponding eigenvectors?

At first glance I assumed that the eigenvectors of $\Sigma$ and $P$ must be the same, but this turns out to be not true. This can be seen by the axes of orientation of the resulting ellipses corresponding to $\Sigma$ and $P$:

library(ellipse)
Sigma <- matrix(c(2, 1.9, 1.9, 5), nrow = 2)
P <- cov2cor(Sigma)

plot(ellipse(Sigma), type = 'l')
lines(ellipse(P), col = "red")

enter image description here

The most I've been able to conclude about the eigenvalues is by using the covariance-correlation decomponsition: $$ \text{det}(\Sigma) = \left(\prod_{i=1}^{n} \sigma_i^2\right) \text{det}(P)\,, $$ where $\sigma_i^2$ i the $i$th diagonal of $\Sigma$.

$\endgroup$
5
  • $\begingroup$ Just to confirm, by "correlation matrix", do you mean the matrix $E[xx^T]$? $\endgroup$
    – mhdadk
    Apr 23 at 12:01
  • $\begingroup$ @mhdadk no, that would just be the covariance matrix for centered random variables. I mean this. $\endgroup$ Apr 23 at 12:06
  • 2
    $\begingroup$ Since $P = \text{diag}(\Sigma)^{-1/2} \Sigma \text{diag}(\Sigma)^{-1/2}$, then $P$ and $\Sigma$ are congruent. Then, according to Sylvester's law of intertia, $P$ and $\Sigma$ have the same number of positive, negative, and zero eigenvalues. I'm not sure if there is anything more to say. $\endgroup$
    – mhdadk
    Apr 23 at 12:15
  • $\begingroup$ @mhdadk thanks! I didn't know about this general result, although it's specific application to correlation and covariance matrices I think I knew. $\endgroup$ Apr 23 at 12:19
  • $\begingroup$ The first question is readily answered by construction: given two sets of eigenvalues, each containing the same number of nonzero eigenvalues, it's easy to match the nonzero eigenvalues in any order you wish and convert one set to the other through suitable scaling. Thus if one is the set of eigenvalues of a correlation matrix, the other set is the eigenvalues of a suitable covariance matrix. The second question is the non-trivial part: given a set of eigenvectors of a correlation matrix, what are the possible eigenvectors of covariance matrices with those correlations? $\endgroup$
    – whuber
    Apr 23 at 15:37

3 Answers 3

7
$\begingroup$

If $\Sigma$ is diagonal (with arbitrary eigenvalues) then $P$ is just the unit matrix (all eigenvalues equal to one), so there cannot be any general relation between the eigenvalues of $\Sigma$ (alone) to those of $P$.

Also notice that if $\Sigma$ is 2-dimensional then $P$ has the form

$$ P = \begin{pmatrix} 1 & \rho \\ \rho & 1 \end{pmatrix} $$ whose eigenvectors are always $(1,1)$ and $(1,-1)$ regardless of $\rho$:

$$ \begin{pmatrix} 1 & \rho \\ \rho & 1 \end{pmatrix} \begin{pmatrix} 1 \\ 1 \end{pmatrix} = (1+\rho) \begin{pmatrix} 1 \\ 1 \end{pmatrix}$$

$$ \begin{pmatrix} 1 & \rho \\ \rho & 1 \end{pmatrix} \begin{pmatrix} 1 \\ -1 \end{pmatrix} = (1-\rho) \begin{pmatrix} 1 \\ -1 \end{pmatrix}$$

so clearly the eigenvectors of $\Sigma$ cannot be related to those of $P$ either. (Of course given both the eigenvalues and eigenvectors of $\Sigma$, one can determine $\Sigma$, and therefore $P$, and therefore the eigenvalues and eigenvectors of $P$).

$\endgroup$
2
  • $\begingroup$ Ah yes, this makes sense. The point about eigenvectors is quite convincing. Regarding the eigenvalues, the eigenvalues themselves may not be connected, but it seems intuitive that given information about the marginal variances and the correlation matrix eigenvalues, I should be able to get the eigenvalues of the covariance matrix. I don't know why I feel I like should be able to do this, but it seems reasonable, no? $\endgroup$ Apr 24 at 3:58
  • 1
    $\begingroup$ I'm not so sure about it. To get the diagonal elements from the eigenvalues generally requires knowing the full orthogonal transformation which has $n(n-1)/2$ degrees of freedom, while the eigenvalues of $P$ give only $n-1$. (the minus one is because the trace is constant). So I doubt if this will work for $n>2$. $\endgroup$
    – J. Delaney
    Apr 24 at 7:23
9
$\begingroup$

Expanding on my comment:

Since $P = \text{diag}(\Sigma)^{-1/2} \Sigma \text{diag}(\Sigma)^{-1/2}$, where $\text{diag}(\Sigma)$ is the diagonal matrix obtained by considering only the diagonal entries of $\Sigma$, then $P$ and $\Sigma$ are congruent. Then, according to Sylvester's law of intertia, $P$ and $\Sigma$ have the same number of positive, negative, and zero eigenvalues. because both $P$ and $\Sigma$ have the same rank, then they have the same number of zero eigenvalues. Moreover, because they are both positive semi-definite, they will also have the same number of positive eigenvalues (thanks to @whuber for pointing this out).

One more thing: if we let $P = Q_P\Lambda_PQ_P^T$ and $\Sigma = Q_\Sigma\Lambda_\Sigma Q_\Sigma^T$ be the eigendecompositions of $P$ and $\Sigma$ respectively, we can relate their eigenvalues as follows: \begin{align} P &= \text{diag}(\Sigma)^{-1/2} \Sigma \text{diag}(\Sigma)^{-1/2} \\ Q_P\Lambda_PQ_P^T &= \text{diag}(\Sigma)^{-1/2} Q_\Sigma\Lambda_\Sigma Q_\Sigma^T \text{diag}(\Sigma)^{-1/2} \\ \Lambda_P &= Q_P^T \text{diag}(\Sigma)^{-1/2} Q_\Sigma\Lambda_\Sigma Q_\Sigma^T \text{diag}(\Sigma)^{-1/2} Q_P \\ \Lambda_P &= B\Lambda_\Sigma B^T \end{align} where $$B = Q_P^T \text{diag}(\Sigma)^{-1/2} Q_\Sigma$$

$\endgroup$
3
  • 4
    $\begingroup$ You don't need this law of inertia: of course there are no negative eigenvalues and of course the two matrices have the same rank. Therefore the numbers of zero eigenvalues are the same, whence so are the numbers of positive eigenvalues. $\endgroup$
    – whuber
    Apr 23 at 15:33
  • $\begingroup$ @whuber you are correct. Added this in. $\endgroup$
    – mhdadk
    Apr 23 at 16:04
  • $\begingroup$ @mhdadk the second part of your answer does clearly give a relationship the eigenvalues, but doesn't seem that useful. Maybe it is what it is! $\endgroup$ Apr 24 at 3:59
4
$\begingroup$

First question: given a vector $\lambda = (\lambda_1,\ldots,\lambda_n)$ of eigenvalues of a variance matrix $\Sigma,$ what are the possible eigenvalues $(\tau_1, \ldots, \tau_n)$ of a covariance matrix $P$ having correlation matrix $\Sigma$?

Writing the diagonal elements of $\Sigma$ as $\Sigma_{ii} = \sigma_i^2$ (for nonnegative square roots $\sigma_i$), we need to know that the relationship between the matrices is

$$\operatorname{Diag}(\sigma)P\operatorname{Diag}(\sigma) = \Sigma.\tag{*}$$

To interpret this question, I suppose you know the eigenvalues but you don't know $\Sigma.$ Thus, it's possible $\Sigma$ is diagonal, $\Sigma = \operatorname{Diag}(\lambda).$ Then all the matrices in $(*)$ are diagonal and it reduces to $n$ simultaneous equations

$$\sigma_i \tau_i \sigma_i = \lambda_i,$$

for which we find the unique solution

$$\sigma_i = \sqrt{\frac{\lambda_i}{\tau_i}}$$

when all the $\tau_i\gt 0.$ When $\tau_i=0,$ necessarily $\lambda_i=0,$ too, and $\sigma_i$ can be any positive number.

Consequently, the zeros of $\tau$ must match the zeros of $\lambda$ and otherwise the components of $\tau$ can be anything.

Second question: given an orthonormal frame $\mathbf e_1, \ldots, \mathbf e_n$ of eigenvectors of a covariance matrix $\Sigma,$ what are the possible frames of eigenvectors of its correlation matrix $P$?

Again, to interpret this reasonably we must suppose the frame is known but $\Sigma$ is not known.

A dimension-counting argument provides insight. There are only $n$ unknown parameters $\sigma_i$ connecting $\Sigma$ and $P.$ Consequently, the dimension of the manifold of frames of eigenvectors associated with $P$ can be at most $n.$ Because the dimension of the manifold of all orthonormal frames, $\mathbb F_n,$ is $(n-1)+(n-2)+\cdots+1=n(n-1)/2,$ when $n\gt 3$ the possible frames of $\Sigma$ cannot be all possible frames: there is a definite restriction.

There appears to be no simple way of characterizing that restriction, because the map

$$\phi_\Sigma : \mathbb R_+^n \to \mathbb F_n$$

that sends $\sigma$ to the frame of eigenvectors of $P(\sigma,\Sigma) = \operatorname{Diag}(\sigma^{-1})\Sigma \operatorname{Diag}(\sigma^{-1})$ depends on $\Sigma.$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.