1
$\begingroup$

If $\mathbf{x} \sim N(\mathbf{0,I})$ then $\mathbf{AA}^T$ is the covariance matrix of $\mathbf{y} = \mathbf{Ax}$, but what does $\mathbf{A}^T \mathbf{A}$ represent?

In some places I have seen statements like "if $\mathbf{X}$ is the data, then $\mathbf{X}^T \mathbf{X}$ is (proportional/related to) the covariance matrix of the data." But here $\mathbf{A}$ is not data, it is a matrix of coefficients.

I was dealing with a problem where I had to reduce the size of $\mathbf{y}$ from $n$ to $1$ and it was solved via a PCA for $\mathbf{A}^T \mathbf{A}$ such that the reduce one-dimensional $y^*$ is given by $y^* = \mathbf{b}^T\mathbf{x}$ where $\mathbf{b}$ is the (normalised) eigenvector corresponding to the largest eigenvalue of $\mathbf{A}^T \mathbf{A}$.

$\endgroup$
  • 1
    $\begingroup$ Is there a reason to expect it has any meaningful interpretation? $\endgroup$ – shimao May 15 '18 at 9:17
  • $\begingroup$ @shimao Thank you for your comment. I was dealing with a problem where I had to reduce the size of $\mathbf{y}$ from $n$ to $1$ and it was solved via a PCA for $\mathbf{A}^T \mathbf{A}$ such that the reduce one-dimensional $y^*$ is given by $y^* = \mathbf{b}^T\mathbf{x}$ where $\mathbf{b}$ is the (normalised) eigenvector of corresponding to the largest eigenvalue of $\mathbf{A}^T \mathbf{A}$ . (I will add this to the post). $\endgroup$ – Confounded May 15 '18 at 9:23
  • $\begingroup$ This is just a way to compute the 1st Principal Component of $AA^T$. $\endgroup$ – Elvis May 15 '18 at 9:32
  • $\begingroup$ @Elvis Thank you for your comment. $A$ is an $n \times m$ matrix so that the dimensions of $A^TA$ ($m \times m$) and $AA^T$ ($n \times n$) are not the same, so I don't think that the PCAs of the two are the same. $\endgroup$ – Confounded May 15 '18 at 9:36
  • $\begingroup$ The PCA's ain't the same but there's a duality. The eigenvalues of $A^T A$ are the loadings for the PCs of $AA^T$. $\endgroup$ – Elvis May 15 '18 at 9:38
0
$\begingroup$

Expansion of the term

Since, based on your comments, $\mathbf{A}$ has dimensions $n*m$ $\mathbf{A}^T \mathbf{A}$ expands to: $$ A_{1,1}*A_{1,1} + …A_{n,1}*A_{n,1} ,... A_{1,1}*A_{1,m} + …A_{n,1}*A_{n,m}$$ $$ A_{1,m}*A_{1,1} + …A_{n,m}*A_{n,1} ,... A_{1,m}*A_{1,m} + …A_{n,m}*A_{n,m}$$

Interpretation of the expansion

The diagonal is the sum of squares for the coefficients of your regression, so the larger the magnitude of the contribution a coefficient is to your relationship, the larger the value will be.

Limitations on the interpretation

Initially this sounds like a useful way to gauge the importance of each variable to the regression. What this actually means will depend heavily on your data types and pretreatment of variable prior to the regression and so is not usually used to assess variable importance. There is a lot of literature out there on variable importance and contribution.

https://www.sciencedirect.com/science/article/pii/S0951832015001672

https://aichamp.wordpress.com/2017/03/26/variable-importance-and-how-it-is-calculated/

http://blog.minitab.com/blog/adventures-in-statistics-2/how-to-identify-the-most-important-predictor-variables-in-regression-models

$\endgroup$
  • $\begingroup$ Thank you for you reply. In my situation, actually, all rows of $A$ are normalised to be unit vectors, so the diagonal of $A^TA$ is 1. So, $A^TA$ looks like a correlation matrix but it is not clear of what. $\endgroup$ – Confounded May 16 '18 at 9:58
  • $\begingroup$ Could you put a bit more info in your question? What regression method are you using? I suspect that PCR should give very different results here compared to PLS. Could you provide some details of what you are seeing, perhaps a toy example or clips of your data. Also, could you include details such as % variance explained in x and in y and % covariance explained between x and y (basically correlation of your regression transformed to the same units as the other data)? $\endgroup$ – ReneBt May 16 '18 at 10:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.