First time question asker here! Thanks in advance for any suggestions! So here's the issue. I start with a matrix Z which is samples by features. I create a covariance matrix $A = Z^T Z$. Then I perform eigenvalue decomposition on A, so $A = QDQ^T$. My goal is then to solve for Z in terms of Q and D. Here's the algebra showing how I solve for z which results in:

$$Z = \sqrt(D)Q^T$$

*This post (https://math.stackexchange.com/questions/2858299/solve-xtx-a-for-x/2859023) confirms that this math checks out, at least I think.

The problem comes when I try to code the above as a toy example in R

#1. Create toy data
z <- matrix(c(1,2,3,4), nrow = 2)

#2. Calculate symmetric matrix A
A <- t(z) %*% z

#3. Eigenvalue decomposition
ev <- eigen(A)
Q <- ev$vectors
l <- ev$values
D <- diag(l)

#4. Re-multiply to obtain A again, as a sanity check
A_new <- Q %*% D %*% t(Q) # Should equal A

#5. Get expression for z
z_new = sqrt(D) %*% t(Q)  # Should equal z

A_new should equal A, and it does

A and A_new:

5 11
11 25

BUT I also think z_new should equal z, and it does not!


1 3
2 4


2.2108796 4.9978076
-0.3346813 0.1480529

Why doesn't Z == Z_new?! Is the math wrong? Is the code wrong? I've never been so stumped!

Ultimately, what I'm trying to do is subtract out the variance that is explained by the first principal component from the original dataset Z. So my plan is to set the first eigenvalue (entry 0,0 of the D matrix) to 0, and then multiply back out and resolve for Z. If there's an easier way to do that directly, or if my method is

Thank you all

  • 1
    $\begingroup$ You're calculating something like a square root, and they are not unique. Imagine Z is a scalar: then A = Z^2, so both +Z and -Z would be solutions. With matrices there are even more solutions, e.g. you can change the signs of any the D elements without affecting the results. $\endgroup$ – user2554330 Dec 12 '20 at 10:36
  • 1
    $\begingroup$ You got a reply in a comment on SO. If you have a (not the) square root $Z^{1/2}$ of $Z$ then $Z^{1/2} U$ is another square root for any orthogonal matrix $U$. $\endgroup$ – Stéphane Laurent Dec 13 '20 at 13:13
  • $\begingroup$ ah, I see the issue, thank you. $\endgroup$ – kiti15237 Dec 14 '20 at 16:24

Seems like the premise for $Z$ does not correspond to the answer on math stackexchange that you refer to: the idea is to start from $A$ and decompose it into $Z^TZ$. Here, you define $Z$ first ($Z$ corresponds to $X$ in math stackexchange), then calculate $A$ based on $Z$ and then again decompose $A$ into $Z^TZ$. There's a circulatory reference here.

Nevertheless, z and z_new both satisfy the decomposition $A=Z^TZ=Z_{new}^TZ_{new}$, which leads to the conclusion that $Z$ is not unique, i.e. for a $Z$ that satisfies the decomposition, there is a $Z^*=BZ$ that also satisfies it. I'm not sure about the theory on this (if it is generally the case or not). EDIT: comment of @Stephane Laurent on OP indicates that it holds generally for a matrix root.

In R, after your code:

B = solve(z,z_new)
z_new %*% B # is equal to z
  • $\begingroup$ thank you, it makes sense! Follow up question: order doesn't seem to matter, but why doesn't order matter? You could do B = solve(z, z_new) or B = solve (z_new, z), and still z_new %*% B = z. Is that always true? $\endgroup$ – kiti15237 Dec 14 '20 at 16:27
  • $\begingroup$ Well, let's see. That would mean that $ZB=Z_n$ and $Z_nB=Z$ are both true. Solving the first for $B$ yields $Z=Z_nB^{-1}$, hence it must hold that $B=B^{-1}$, i.e. $B$ is an involutory matrix. This is the case for the matrix in your example, but does not generally hold (basically coincidence). Try for example replacing 4 with 7 in your matrix, you'll get different results. $\endgroup$ – PaulG Dec 14 '20 at 18:48
  • $\begingroup$ *solving the first for $Z$ $\endgroup$ – PaulG Dec 15 '20 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.