Hastie "statistical learning" 2.28. Least squares and covariance

Question

In Hasties book "statistical learning", just above equation 2.28, it says that $\mathbf{X}^T\mathbf{X} \rightarrow NCov(X)$ (when $N$ is large and $E(X)=0$).

Why is this true?

$Cov(X)$ is obviously $Cov(X,X)$, and since $Cov(X,X) = Var(X)$ and $E(X)=0$, $Cov(X)=E(X^2)$. So the original equation says that $\mathbf{X}^T\mathbf{X}\rightarrow NE(X^2)$.

But, why is $\mathbf{X}^T\mathbf{X}$ scalar? Shoudn't this be a p x p matrix? Where p is the dimensionality.

This is a book that requires a lot of work, apperently...

Answer 1

$X = (X_1, \ldots, X_p)$ is a random vector with $p$ entries (not a scalar random variable).

The expectation of this random vector is denoted $E[X] = (E[X_1], \ldots, E[X_p])$.

The covariance matrix of this random vector is denoted $\text{Cov}(X)$; it is a $p \times p$ matrix whose $(i, j)$ entry is $\text{Cov}(X_i, X_j)$.

$\mathbf{X}$ is an $N \times p$ matrix where each row is an i.i.d. draw from the distribution of the random vector $X$.

The sample covariance matrix $\frac{1}{N} \mathbf{X}^\top \mathbf{X}$ is a $p \times p$ matrix whose $(i, j)$ entry is $\frac{1}{N} \sum_{n=1}^N X_{n, i} X_{n, j}$. For each $n$, we have $E[X_{n, i} X_{n, j}] = \text{Cov}(X_i, X_j)$, so $\frac{1}{N} \sum_{n=1}^N X_{n, i} X_{n, j}$ tends to $\text{Cov}(X_i, X_j)$ by the law of large numbers.