For a given data matrix $A$ (with variables in columns and data points in rows), it seems like $A^TA$ plays an important role in statistics. For example, it is an important part of the analytical solution of ordinary least squares. Or, for PCA, its eigenvectors are the principal components of the data.
I understand how to calculate $A^TA$, but I was wondering if there's an intuitive interpretation of what this matrix represents, which leads to its important role?
Geometrically, matrix $\bf A'A$ is called matrix of scalar products (= dot products, = inner products). Algebraically, it is called sum-of-squares-and-cross-products matrix (SSCP).
Its $i$-th diagonal element is equal to $\sum a_{(i)}^2$, where $a_{(i)}$ denotes values in the $i$-th column of $\bf A$ and $\sum$ is the sum across rows. The $ij$-th off-diagonal element therein is $\sum a_{(i)}a_{(j)}$.
There is a number of important association coefficients and their square matrices are called angular similarities or SSCP-type similarities:
Dividing SSCP matrix by $n$, the sample size or number of rows of $\bf A$, you get MSCP (mean-square-and-cross-product) matrix. The pairwise formula of this association measure is hence $\frac{\sum xy}{n}$ (with vectors $x$ and $y$ being a pair of columns from $\bf A$).
If you center columns (variables) of $\bf A$, then $\bf A'A$ is the scatter (or co-scatter, if to be rigorous) matrix and $\mathbf {A'A}/(n-1)$ is the covariance matrix. Pairwise formula of covariance is $\frac{\sum c_xc_y}{n-1}$ with $c_x$ and $c_y$ denoting centerted columns.
If you z-standardize columns of $\bf A$ (subtract the column mean and divide by the standard deviation), then $\mathbf {A'A}/(n-1)$ is the Pearson correlation matrix: correlation is covariance for standardized variables. Pairwise formula of correlation is $\frac{\sum z_xz_y}{n-1}$ with $z_x$ and $z_y$ denoting standardized columns. The correlation is also called coefficient of linearity.
If you unit-scale columns of $\bf A$ (bring their SS, sum-of-squares, to 1), then $\bf A'A$ is the cosine similarity matrix. The equivalent pairwise formula thus appears to be $\sum u_xu_y = \frac{\sum{xy}}{\sqrt{\sum x^2}\sqrt{\sum y^2}}$ with $u_x$ and $u_y$ denoting L2-normalized columns. Cosine similarity is also called coefficient of proportionality.
If you center and then unit-scale columns of $\bf A$, then $\bf A'A$ is again the Pearson correlation matrix, because correlation is cosine for centered variables$^{1,2}$: $\sum cu_xcu_y = \frac{\sum{c_xc_y}}{\sqrt{\sum c_x^2}\sqrt{\sum c_y^2}}$
Alongside these four principal association measures let us also mention some other, also based on of $\bf A'A$, to top it off. They can be seen as measures alternative to cosine similarity because they adopt different from it normalization, the denominator in the formula:
Coefficient of identity [Zegers & ten Berge, 1985] has its denominator in the form of arithmetic mean rather than geometric mean: $\frac{\sum{xy}}{(\sum x^2+\sum y^2)/2}$. It can be 1 if and only if the being compared columns of $\bf A$ are identical.
Another usable coefficient like it is called similarity ratio: $\frac{\sum{xy}}{\sum x^2 + \sum y^2 -\sum {xy}} = \frac{\sum{xy}}{\sum {xy} + \sum {(x-y)^2}}$.
Finally, if values in $\bf A$ are nonnegative and their sum within the columns is 1 (e.g. they are proportions), then $\bf \sqrt {A}'\sqrt A$ is the matrix of fidelity or Bhattacharyya coefficient.
$^1$ One way also to compute correlation or covariance matrix, used by many statistical packages, bypasses centering the data and departs straight from SSCP matrix $\bf A'A$ this way. Let $\bf s$ be the row vector of column sums of data $\bf A$ while $n$ is the number of rows in the data. Then (1) compute the scatter matrix as $\bf C = A'A-s's/ \it n$ [thence, $\mathbf C/(n-1)$ will be the covariance matrix]; (2) the diagonal of $\bf C$ is the sums of squared deviations, row vector $\bf d$; (3) compute correlation matrix $\bf R=C/\sqrt{d'd}$.
$^2$ An acute but statistically novice reader might find it difficult reconciling the two definitions of correlation - as "covariance" (which includes averaging by sample size, the division by df="n-1") and as "cosine" (which implies no such averaging). But in fact no real averaging in the first formula of correlation takes place. The thing is that st. deviation, by which z-standardization was achieved, had been in turn computed with the division by that same df; and so the denominator "n-1" in the formula of correlation-as-covariance entirely cancels if you unwrap the formula: the formula turns into the formula of cosine. To compute empirical correlation value you really need not to know $n$ (except when computing the mean, to center).
The matrix $A^TA$ contains all the inner products of all columns in $A$. The diagonal thus contains the squared norms of columns. If you think about geometry and orthogonal projections onto the column space spanned by the columns in $A$ you may recall that norms and inner products of the vectors spanning this space play a central role in the computation of the projection. Least squares regression as well as principal components can be understood in terms of orthogonal projections.
Also note that if the columns of $A$ are orthonormal, thus forming an orthonormal basis for the column space, then $A^TA = I$ $-$ the identity matrix.
@NRH gave a good technical answer.
If you want something really basic, you can think of $A^TA$ as the matrix equivalent of $A^2$ for a scalar.
An important view of the geometry of $A'A$ is this (the viewpoint strongly stressed in Strang's book on "Linear Algebra and Its Applications"): Suppose A is an $m \times n$-matrix of rank k, representing a linear map $A: R^n \rightarrow R^m$. Let Col(A) and Row(A) be the column and row spaces of $A$. Then
(a) As a real symmetric matrix, $(A'A): R^n \rightarrow R^n$ has a basis $\{e_1,..., e_n\}$ of eigenvectors with non-zero eigenvalues $d_1,\ldots,d_k$. Thus:
$(A'A)(x_1e_1 + \ldots + x_ne_n) = d_1x_1e_1 + ... + d_kx_ke_k$.
(b) Range(A) = Col(A), by definition of Col(A). So A|Row(A) maps Row(A) into Col(A).
(c) Kernel(A) is the orthogonal complement of Row(A). This is because matrix multiplication is defined in terms of the dot products (row i)*(col j). (So $Av'= 0 \iff \text{v is in Kernel(A)} \iff v \text{is in orthogonal complement of Row(A)}$
(d) $A(R^n)=A(\text{Row}(A))$ and $A|\text{Row(A)}:\text{Row(A)} \rightarrow Col(A)$ is an isomorphism.
Reason: If v = r+k (r \in Row(A), k \in Kernel(A),from (c)) then
A(v) = A(r) + 0 = A(r) where A(r) = 0 <==> r = 0$.
[Incidentally gives a proof that Row rank = Column rank!]
(e) Applying (d), $A'|:Col(A)=\text{Row(A)} \rightarrow \text{Col(A')}=\text{Row(A)}$ is an isomorphism
(f)By (d) and (e): $A'A(R^n) = \text{Row(A)}$ and A'A maps Row(A) isomorphically onto Row(A).
Although it has been already discussed that $\textbf{A}^T\textbf{A}$ has the meaning of taking dot products, I would only add a graphical representation of this multiplication.
Indeed, while rows of the matrix $\textbf{A}^T$ (and columns of the matrix $\textbf{A}$) represent variables, we treat each variable measurements as a multidimensional vector. Multiplying the row $p$ of $\textbf{A}^T$ with the column $p$ of $\textbf{A}$ is equivalent to taking the dot product of two vectors: $dot(A_{T, row_p}, A_{col_p})$.
Similarly, multiplying the row $p$ of $\textbf{A}^T$ with the column $k$ of $\textbf{A}$ is equivalent to the dot product: $dot(A_{T, row_p}, A_{col_k})$.
The element $(p, k)$ of the resulting matrix $\textbf{A}^T\textbf{A}$ has the meaning of how much the vector $A_{T, row_p}$ is in the direction of the vector $A_{col_k}$. If the dot product of two vectors $A_{T, row_i}$ and $A_{col_j}$ is other than zero, some information about a vector $A_{T, row_i}$ is carried by a vector $A_{col_j}$, and vice versa.
This idea plays an important role in Principal Component Analysis, where we want to find a new representation of our initial data matrix $\textbf{A}$ such that, there is no more information carried about any column $i$ in any other column $j \neq i$.
There are levels of intuition. For those familiar with matrix notation instatistics the intuition is to think of it as a square of the random variable: $x\to E[x^2]$ vs $A\to A^TA$
In matrix notation a sample of the random variable $x$ observations $x_i$ or a population are represented by a column vector: $$a=\begin{bmatrix} x_1 \\ x_2 \\ \dots \\ x_n \end{bmatrix}$$
So, if you want to get a sample mean of the square of the variable $x$, you simply get a dot product $$\bar{x^2}=\frac{a\cdot a} n$$, which is the same in matrix notation as $A^TA$.
Notice, that if the sample mean of the variable is ZERO, then the variance is equal to the mean of the square: $\sigma^2=E[x^2]$ which is analogous to $A^TA$. This is the reason why in PCA you need the zero mean, and why $A^TA$ shows up, after all PCA is to decompose the variance matrix of the data set.
