# interpretation of $A^T A$ [duplicate]

Is there a statistical/information theoretic interpretation of this matrix in the context where A represents observations of data (and this matrix which shows up in e.g. solving linear systems and elsewhere)? Does it have a special name used in the context where A represents "rows of data"?

• When the variables in $A$ are centered, $A^TA/(n-1)$ is the variance-covariance matrix of the variables.
– Noah
Jun 19, 2020 at 1:39

When $$A$$ represents observations of data as you mentioned, it is typically called a design matrix. As for a special name for $$A'A$$, I have never encountered a popular one, though I have heard "the design matrix transposed times itself," which is hardly simple and nice. As for statistical interpretation, as Noah mentioned in the comment to your question, if each row $$i$$ of $$A\in\mathbb{R}^{n\times n}$$ is $$(A_{i1},\dots,A_{ik})$$, then $$A'A = nCov$$, where $$Cov$$ is the covariance matrix of $$a$$, where the $$i,j$$-th entry corresponds to $$Cov(A_i,A_j)$$ (this can be verified manually by simply performing the matrix multiplication).
As you mentioned, a classic appearance it makes is in the famous equation when trying to find the $$\hat{\beta}$$ that minimizes the sum of squared residuals in a linear regression $$y = A\beta + e$$, where we have that $$A'A\hat{\beta} = A'y \implies \hat{\beta} = (A'A)^{-1}X'y$$
with the last equality assuming that $$A$$ is linearly independent (and has more observations than parameters).
Since it shows up in linear regression, it also shows up when calculating variance of $$\hat{\beta}$$. The simplest (and most common) case is under Gauss-Markov assumptions, in which case you can show that
$$Var(\hat{\beta}) = \sigma^2(A'A)^{-1}$$
What role does it play here? One explanation is that if the spread of $$A$$ increases, $$(A'A)^{-1}$$ decreases in magnitude, thus reducing the variance of the estimator. Intuitively, if you have more spread out observations, you get a better estimator. It will also show up for many tests surrounding linear regressions, including, for example, tests of coefficients being equal (for example, testing that $$\beta_1 = \beta_2$$).