# If $\mathbf{x} \sim N(\mathbf{0,I})$ and $\mathbf{y} = \mathbf{Ax}$, what does $\mathbf{A}^T \mathbf{A}$ represent?

If $\mathbf{x} \sim N(\mathbf{0,I})$ then $\mathbf{AA}^T$ is the covariance matrix of $\mathbf{y} = \mathbf{Ax}$, but what does $\mathbf{A}^T \mathbf{A}$ represent?

In some places I have seen statements like "if $\mathbf{X}$ is the data, then $\mathbf{X}^T \mathbf{X}$ is (proportional/related to) the covariance matrix of the data." But here $\mathbf{A}$ is not data, it is a matrix of coefficients.

I was dealing with a problem where I had to reduce the size of $\mathbf{y}$ from $n$ to $1$ and it was solved via a PCA for $\mathbf{A}^T \mathbf{A}$ such that the reduce one-dimensional $y^*$ is given by $y^* = \mathbf{b}^T\mathbf{x}$ where $\mathbf{b}$ is the (normalised) eigenvector corresponding to the largest eigenvalue of $\mathbf{A}^T \mathbf{A}$.

• Is there a reason to expect it has any meaningful interpretation? – shimao May 15 '18 at 9:17
• @shimao Thank you for your comment. I was dealing with a problem where I had to reduce the size of $\mathbf{y}$ from $n$ to $1$ and it was solved via a PCA for $\mathbf{A}^T \mathbf{A}$ such that the reduce one-dimensional $y^*$ is given by $y^* = \mathbf{b}^T\mathbf{x}$ where $\mathbf{b}$ is the (normalised) eigenvector of corresponding to the largest eigenvalue of $\mathbf{A}^T \mathbf{A}$ . (I will add this to the post). – Confounded May 15 '18 at 9:23
• This is just a way to compute the 1st Principal Component of $AA^T$. – Elvis May 15 '18 at 9:32
• @Elvis Thank you for your comment. $A$ is an $n \times m$ matrix so that the dimensions of $A^TA$ ($m \times m$) and $AA^T$ ($n \times n$) are not the same, so I don't think that the PCAs of the two are the same. – Confounded May 15 '18 at 9:36
• The PCA's ain't the same but there's a duality. The eigenvalues of $A^T A$ are the loadings for the PCs of $AA^T$. – Elvis May 15 '18 at 9:38

## Expansion of the term

Since, based on your comments, $\mathbf{A}$ has dimensions $n*m$ $\mathbf{A}^T \mathbf{A}$ expands to: $$A_{1,1}*A_{1,1} + …A_{n,1}*A_{n,1} ,... A_{1,1}*A_{1,m} + …A_{n,1}*A_{n,m}$$ $$A_{1,m}*A_{1,1} + …A_{n,m}*A_{n,1} ,... A_{1,m}*A_{1,m} + …A_{n,m}*A_{n,m}$$

## Interpretation of the expansion

The diagonal is the sum of squares for the coefficients of your regression, so the larger the magnitude of the contribution a coefficient is to your relationship, the larger the value will be.

## Limitations on the interpretation

Initially this sounds like a useful way to gauge the importance of each variable to the regression. What this actually means will depend heavily on your data types and pretreatment of variable prior to the regression and so is not usually used to assess variable importance. There is a lot of literature out there on variable importance and contribution.

https://www.sciencedirect.com/science/article/pii/S0951832015001672

https://aichamp.wordpress.com/2017/03/26/variable-importance-and-how-it-is-calculated/

• Thank you for you reply. In my situation, actually, all rows of $A$ are normalised to be unit vectors, so the diagonal of $A^TA$ is 1. So, $A^TA$ looks like a correlation matrix but it is not clear of what. – Confounded May 16 '18 at 9:58