Why does the variance of the OLS estimator scale with $\frac{1}{n}$?

Question

Let $\mathbf{X}$ be an $n\times p$ matrix. In multiple linear regression, we have

$$\boldsymbol{\hat{\beta}}=\mathbf{[X^TX]^{-1}X^Ty}\sim\mathcal{N}\Big(\boldsymbol{\beta}, \sigma^2 \mathbf{[X^TX]^{-1}}\Big)$$

I have read $\mathbf{[X^TX]^{-1}}$ grows with $\frac{1}{n}$, but I don't understand why.

For whatever reason, $\exists \mathbf{A}$ such that

$$\mathbf{X^TX}\approx n \mathbf{A}$$ where $\mathbf{A}$ is a constant matrix.

Suppose the rows of $\mathbf{X}$ are i.i.d. from some distribution who's random variable maps to $\mathbb{R}^p$. Then $(\mathbf{X^TX})_{ij} = \mathbf{x}_i \mathbf{x}_j^T$.

Somehow this helps us see the matrix $\mathbf{X^TX}$ is positive semi-definite.

Somehow knowing that helps us see that, by the law of large numbers, $$\mathbf{X^TX}\approx n \mathbb{E}[\mathbf{x}_i\mathbf{x}_j]^T$$

The fact that $\text{rank}(\mathbf{x}_i)=1$ is fine because, after taking the expected value, $\mathbf{X^TX}$ will be full rank.

I didn't really follow this proof. Could someone elaborate and fill in the gaps in my understanding?

Answer 1

Intuition:

Imagine that $X$ is only a single column relating to the intercept (estimating the mean of a distribution). Then it will be a vector of size $n $ with only ones and $X^tX=n $.

For $X $ a matrix with more columns it goes a bit the same. Each cell in the matrix $X^tX $ is the product of two vector of size $n $ and scales with $n $.

Sort of, it depends (of course) what numbers make up $X $, but if you double or triple it in size, that is the same rows/numbers now occur twice or three times as much, like taking duplicate or triplicate measurements, then you will get that $X^tX$ grows as $n $.

For other types of increase of the sample size $n $ you do not need to get that $X^tX $ grows as $n $.

Example:

$$\begin{bmatrix} x_{11} &x_{12} &x_{11} &x_{12} \\ x_{21} &x_{22} &x_{21} &x_{22} \\ \end{bmatrix} \times \begin {bmatrix} x_{11} & x_{21} \\ x_{12} & x_{22} \\ x_{11} & x_{21} \\ x_{12} & x_{22} \\ \end {bmatrix} = 2 \cdot \begin{bmatrix} x_{11} &x_{12} \\ x_{21} &x_{22} \\ \end{bmatrix} \times \begin {bmatrix} x_{11} & x_{21} \\ x_{12} & x_{22} \\ \end {bmatrix} $$

but when the growth in $n $ is adding different points then most often you do not scale with $n $.

$$\begin{bmatrix} x_{11} &x_{12} &x_{13} &x_{14} \\ x_{21} &x_{22} &x_{23} &x_{24} \\ \end{bmatrix} \times \begin {bmatrix} x_{11} & x_{21} \\ x_{12} & x_{22} \\ x_{13} & x_{23} \\ x_{14} & x_{24} \\ \end {bmatrix} \neq 2 \cdot \begin{bmatrix} x_{11} &x_{12} \\ x_{21} &x_{22} \\ \end{bmatrix} \times \begin {bmatrix} x_{11} & x_{21} \\ x_{12} & x_{22} \\ \end {bmatrix} $$

Adding more sample points may sometimes improve the variance of $\beta$ a lot and sometimes it improves it a little. It depends on which $x$ you add as more sample points. When you repeat the same points then the product $X^tX $ grows as $n $.

Answer 2

Your result is not generally true. For a counterexample, suppose a simple linear regression ($p=2$) and, for simplicity, $n$ even. Suppose that the vector $x$ of predictor values is $x=[-n/2,-n/2+1, \dotsc, n/2]^T$ (but omitting zero). To get the design matrix $\mathbf{X}$ augment a column of ones. Then we can calculate that $$ \mathbf{X}^T \mathbf{X}=\begin{pmatrix} n& 0 \\ 0 & \frac{n(n+1)(n+2)}{12} \end{pmatrix} $$ so $$ (\mathbf{X}^T\mathbf{X})^{-1}=\begin{pmatrix} 1/n& 0 \\ 0 & \frac{12}{n(n+1)(n+2)} \end{pmatrix} $$ and so the variance of the slope estimator will go to zero much faster than $1/n$, while the variance of the intercept estimator will have the usual rate.

So it is clear that for your result to hold, some assumptions must be fulfilled.

One such condition is: suppose that the $x_1, x_2, \dotsc, x_n$ vectors filling up the rows of $\mathbf{X}$ are independent draws from some fixed distribution on $\mathbb{R}^p$. Then $\mathbf{X}^T \mathbf{X} = \sum_1^n x_i x_i^T $ and by taking expectation $\DeclareMathOperator{\E}{\mathbb{E}} \E \mathbf{X}^T \mathbf{X}=\sum_1^n \E x_i x_i^T =n \mathbf{A}$, say. For the application of the law of large numbers we must assume that that expectation really exists, so the assumed common distribution must have existing mean and variance. For my counterexample, the new design points come in a deterministic, increasing sequence, so these assumptions are not fulfilled, there is no common distribution.

Certainly some weaker assumptions might suffice for a proof.

Answer 3

Let $x_{ik},x_{jk}$ be random distributed, then your $\mathbf{[X^TX]}$ is $\sum_{k=1}^nx_{ik}x_{jk}\to n\times cov[x_i,x_j]$ under usual OLS assumptions. That's how $\sigma^2\mathbf{[X^TX]}^{-1}$ will have a term declining at $\sim\frac 1 n$.

Additionally, you could see how $\mathbf{[X^TX]}\to n\times \Sigma$, where $\Sigma$ is a variance-covariance matrix of the design matrix of regressors. This is why it's good to have a lot of variability in regressor values.