4
$\begingroup$

Let $X$ be a $n\times (p+1)$ non-stochastic design matrix. OLS estimator is given by

$$\hat{\beta} = (X'X)^{-1}X' y$$

Thus the variance of the estimator is

$$\text{Var}\left( \hat{\beta}\right) = (X'X)^{-1} \sigma^2\, , $$

where $\text{Var}(y) = I_n \sigma^2$.

My question is, why is it true that variance of estimator decreases as sample size increases? It is not obvious to me what the $i$-th diagonal entry of $(X'X)^{-1}$ is.

$\endgroup$
  • $\begingroup$ The result isn't necessarily true, because it depends on how $X$ changes with $n$. Could you tell us what you are proposing? $\endgroup$ – whuber Nov 7 '17 at 20:44
  • 1
    $\begingroup$ @whuber I think they are looking for a mathematical proof of std. error of beta hat = variance / sqrt(n)? $\endgroup$ – Mark White Nov 7 '17 at 21:09
  • $\begingroup$ @Mark That's true when the only regressor is a constant, because there's no question about what happens to it as more data are collected! However, in any other case, what values are we to give to each new regressor as we collect more data? We can choose sequences in which the variance of a coefficient estimate goes up for a while before it goes down. We can even choose sequences in which the variance oscillates forever--and never converges to zero! $\endgroup$ – whuber Nov 7 '17 at 21:23
  • 1
    $\begingroup$ I think he's assuming that you could make infinite draws from a totally stationary dataset, and that all draws would be of sufficiently large $n$ that their variance would be the same. $\endgroup$ – Josh Nov 7 '17 at 21:49
  • 1
    $\begingroup$ @Josh That sounds like a particular kind of stochastic design matrix. It's a reasonable interpretation, though, and helps make the desired conclusion true! $\endgroup$ – whuber Nov 7 '17 at 21:56
5
$\begingroup$

Assumptions:

(1) There exists a population from which infinite draws of $X$ and $y$ may be made, and each of those draws are characterized by the exact same distribution parameters.

(2) $n$ is sufficiently large that the variance of a sample of length $n$ is always the same, or may be approximated as such.

Let's start out like this:

$\hatβ=({X'X})^{-1}X'y$

$\text{Var}(\hatβ)=\text{Var}[({X'X})^{-1}X'y]$

Now, let the columns of $X$ be mutually orthogonal, each with variance $σ^2$ and mean $0$. $X'X$ is then a $(p+1)$-dimensional diagonal matrix whose elements are $nσ^2$. $({X'X})^{-1}$ is just the element-by-element inversion of the diagonals of $X'X$, that is, a $(p+1)$-dimensional diagonal matrix whose elements are $1/{(nσ^2)}$.

That brings us to

$\text{Var}(\hatβ)=[1/{(nσ^2)}]^2I_{p+1}\text{Var}[X'y]$

$\text{Var}(\hatβ)=[1/{(n^2σ^4)}]I_{p+1}\text{Var}[X'y]$

However, if $y$ is just a univariate response with variance $σ^2$ and mean $0$, then there's no need for the identity matrix in specifying its variance; its variance is a scalar. As specified in the first paragraph, each of the columns of $X$ also has variance $σ^2$ and mean $0$, so the variance of $X'y$ is given by a $(p+1)$-by-$1$ column vector whose elements are $nσ^4$, i.e., $nσ^41_{p+1}$. The presence of the $n$ term seems strange until you realize that we are actually talking about the variance of the sum of $n$ random variables, each with variance $σ^4$ (the product of two random variables each with variance $σ^2$ and mean $0$). That is,

$\text{Var}(\hatβ)=[1/{(n^2σ^4)}]I_{p+1}nσ^41_{p+1}$

So we have a $(p+1)$-by-$(p+1)$ diagonal matrix multiplying a $(p+1)$-by-$1$ vector, each of whose elements are

$\text{Var}(\hatβ_i)=[1/{(n^2σ^4)}]nσ^4=1/n$

Note the absence of $σ^2$, which is due to our specification that all the vectors have the same variance. The summation of the $p+1$ elements of the variance vector therefore scales linearly with $p+1$, which we also expect. This is essentially the variance of $\hat{y}$, which tends to exhibit proportionality to $(p+1)/n$.

Here is a resource I've found useful, and extends this explanation to regularized (ridge) regression.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ This answer outlines one way in which to make the conclusion true. There are a lot of assumptions here--perhaps it would clarify the exposition if you were to lay them out explicitly. Given that the question specifies that $X$ is "non-stochastic," could you explain the sense(s) in which you are referring to their "variance" in this answer? $\endgroup$ – whuber Nov 7 '17 at 21:54
  • 1
    $\begingroup$ Thank you for undertaking that listing of assumptions. However, it's now almost certain that the columns of $X$ will not be orthogonal! That doesn't ruin your basic idea, but it does indicate that this problem may be more subtle than it might appear. $\endgroup$ – whuber Nov 7 '17 at 21:59
  • 1
    $\begingroup$ Yes--but that's an extra operation you have to undertake and when you do it, the coefficients no longer have the same meaning from one draw to the next. $\endgroup$ – whuber Nov 7 '17 at 22:01
  • 2
    $\begingroup$ (Answering questions on this site is a great way to increase that knowledge!) $\endgroup$ – whuber Nov 7 '17 at 22:16
  • 2
    $\begingroup$ I know you've been in this community for some time, but your recent activity suggests a renewed interest--so welcome! $\endgroup$ – whuber Nov 7 '17 at 22:19
2
$\begingroup$

If we assume that $\sigma^2$ is known, the variance of the OLS estimator only depends on $X'X$ because we do not need to estimate $\sigma^2$. Here is a purely algebraic proof that the variance of the estimator decreases with any additional observation if $\sigma^2$ is known. Suppose $X$ is your current design matrix and you add one more observation $x$, which has dimension $1\times (p+1)$. Your new design matrix is $$X_{new} = \left(\begin{array}{c}X \\ x \end{array}\right).$$ You can check that $X_{new}'X_{new} = X'X + x'x$. Using the Woodbury identity we get $$ (X_{new}'X_{new})^{-1} = (X'X + x'x)^{-1} = (X'X)^{-1} - \frac{(X'X)^{-1}x'x(X'X)^{-1}}{1+xx'} $$ Because $(X'X)^{-1}x'x(X'X)^{-1}$ is positive semi-definite (it is the multiplication of a matrix with its transpose) and $1+x'x>0$, the diagonal elements of the subtracting term are greater than or equal to zero. So, the diagonal elements of $(X_{new}'X_{new})^{-1}$ are less than or equal to the diagonal elements of $(X'X)^{-1}$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1. Thank you for the clear explanation -- and welcome to CV. $\endgroup$ – whuber Jun 24 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.