Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Let $X$ be a $n\times (p+1)$ non-stochastic design matrix. OLS estimator is given by
$$\hat{\beta} = (X'X)^{-1}X' y$$
Thus the variance of the estimator is
$$\text{Var}\left( \hat{\beta}\right) = (X'X)^{-1} \sigma^2\, , $$
where $\text{Var}(y) = I_n \sigma^2$.
My question is, why is it true that variance of estimator decreases as sample size increases? It is not obvious to me what the $i$-th diagonal entry of $(X'X)^{-1}$ is.
If we assume that $\sigma^2$ is known, the variance of the OLS estimator only depends on $X'X$ because we do not need to estimate $\sigma^2$. Here is a purely algebraic proof that the variance of the estimator decreases with any additional observation if $\sigma^2$ is known. Suppose $X$ is your current design matrix and you add one more observation $x$, which has dimension $1\times (p+1)$. Your new design matrix is $$X_{new} = \left(\begin{array}{c}X \\ x \end{array}\right).$$ You can check that $X_{new}'X_{new} = X'X + x'x$. Using the Woodbury identity we get $$ (X_{new}'X_{new})^{-1} = (X'X + x'x)^{-1} = (X'X)^{-1} - \frac{(X'X)^{-1}x'x(X'X)^{-1}}{1+x(X'X)^{-1}x'} $$ Because $(X'X)^{-1}x'x(X'X)^{-1}$ is positive semi-definite (it is the multiplication of a matrix with its transpose) and $1+x(X'X)^{-1}x'>0$, the diagonal elements of the subtracting term are greater than or equal to zero. So, the diagonal elements of $(X_{new}'X_{new})^{-1}$ are less than or equal to the diagonal elements of $(X'X)^{-1}$.
Assumptions:
(1) There exists a population from which infinite draws of $X$ and $y$ may be made, and each of those draws are characterized by the exact same distribution parameters.
(2) $n$ is sufficiently large that the variance of a sample of length $n$ is always the same, or may be approximated as such.
Let's start out like this:
$\hatβ=({X'X})^{-1}X'y$
$\text{Var}(\hatβ)=\text{Var}[({X'X})^{-1}X'y]$
Now, let the columns of $X$ be mutually orthogonal, each with variance $σ^2$ and mean $0$. $X'X$ is then a $(p+1)$-dimensional diagonal matrix whose elements are $nσ^2$. $({X'X})^{-1}$ is just the element-by-element inversion of the diagonals of $X'X$, that is, a $(p+1)$-dimensional diagonal matrix whose elements are $1/{(nσ^2)}$.
That brings us to
$\text{Var}(\hatβ)=[1/{(nσ^2)}]^2I_{p+1}\text{Var}[X'y]$
$\text{Var}(\hatβ)=[1/{(n^2σ^4)}]I_{p+1}\text{Var}[X'y]$
However, if $y$ is just a univariate response with variance $σ^2$ and mean $0$, then there's no need for the identity matrix in specifying its variance; its variance is a scalar. As specified in the first paragraph, each of the columns of $X$ also has variance $σ^2$ and mean $0$, so the variance of $X'y$ is given by a $(p+1)$-by-$1$ column vector whose elements are $nσ^4$, i.e., $nσ^41_{p+1}$. The presence of the $n$ term seems strange until you realize that we are actually talking about the variance of the sum of $n$ random variables, each with variance $σ^4$ (the product of two random variables each with variance $σ^2$ and mean $0$). That is,
$\text{Var}(\hatβ)=[1/{(n^2σ^4)}]I_{p+1}nσ^41_{p+1}$
So we have a $(p+1)$-by-$(p+1)$ diagonal matrix multiplying a $(p+1)$-by-$1$ vector, each of whose elements are
$\text{Var}(\hatβ_i)=[1/{(n^2σ^4)}]nσ^4=1/n$
Note the absence of $σ^2$, which is due to our specification that all the vectors have the same variance. The summation of the $p+1$ elements of the variance vector therefore scales linearly with $p+1$, which we also expect. This is essentially the variance of $\hat{y}$, which tends to exhibit proportionality to $(p+1)/n$.
Here is a resource I've found useful, and extends this explanation to regularized (ridge) regression.
