interpretation of $A^T A$ Is there a statistical/information theoretic interpretation of this matrix in the context where A represents observations of data (and this matrix which shows up in e.g. solving linear systems and elsewhere)? Does it have a special name used in the context where A represents "rows of data"?
 A: When $A$ represents observations of data as you mentioned, it is typically called a design matrix. As for a special name for $A'A$, I have never encountered a popular one, though I have heard "the design matrix transposed times itself," which is hardly simple and nice. As for statistical interpretation, as Noah mentioned in the comment to your question, if each row $i$ of $A\in\mathbb{R}^{n\times n}$ is $(A_{i1},\dots,A_{ik})$, then $A'A = nCov$, where $Cov$ is the covariance matrix of $a$, where the $i,j$-th entry corresponds to $Cov(A_i,A_j)$ (this can be verified manually by simply performing the matrix multiplication).
As you mentioned, a classic appearance it makes is in the famous equation when trying to find the $\hat{\beta}$ that minimizes the sum of squared residuals in a linear regression $y = A\beta + e$, where we have that
$$A'A\hat{\beta} = A'y \implies \hat{\beta} = (A'A)^{-1}X'y$$
with the last equality assuming that $A$ is linearly independent (and has more observations than parameters).
Since it shows up in linear regression, it also shows up when calculating variance of $\hat{\beta}$. The simplest (and most common) case is under Gauss-Markov assumptions, in which case you can show that
$$Var(\hat{\beta}) = \sigma^2(A'A)^{-1}$$
What role does it play here? One explanation is that if the spread of $A$ increases, $(A'A)^{-1}$ decreases in magnitude, thus reducing the variance of the estimator. Intuitively, if you have more spread out observations, you get a better estimator. It will also show up for many tests surrounding linear regressions, including, for example, tests of coefficients being equal (for example, testing that $\beta_1 = \beta_2$).
