Shouldn't the variance of the vector of regression coefficients decrease when we have larger sample sizes?

Question

In the first answer to this post, it is shown that the variance of the estimated regression coefficient $\hat \beta_1$ in simple linear regression is $$ \text{Var}(\hat \beta_1) = \frac{\sigma^2}{\sum_i (x_i - \bar x)^2}. \quad \quad \quad (1) $$

On Wikipedia we have the following expression for the variance-covariance matrix of the vector of coefficients: $$ \text{Var}(\hat \beta|X) = \sigma^2 (X^T X)^{-1}. \quad \quad \quad (2) $$

The expressions (1) and (2) are quite similar. In expression (1), it is clear that for larger samples sizes, the variance will decrease as we will have more terms in the summation in the denominator. But for expression 2, it seems that there is no guarantee that the variance (the diagonals of the covariance matrix) will decrease if we take larger samples sizes?

Also, can expression (1) be obtained from expression (2)?

Edit: The result for $\text{Var}(\hat \beta_1$) in (1) should also be contained in the a corresponding entry on the diagonal of $\text{Var}(\hat \beta | X)$? How do we show that (1) corresponds to an entry on the diagonal of (2)? I'm not seeing where the subtraction of the mean in the denominator of (1) arises in (2)?

Answer 1

In regression analysis, if you want to get asymptotic consistency results you need to impose some limiting conditions on the explanatory variables. The regression model itself makes no assumptions about the form of the sequence of explanatory variables, so these limiting conditions are conditions that go beyond the model assumptions for regression analysis. There are various (sufficient) limiting conditions that can be imposed on the sequence of explanatory variables in order to get consistency of the OLS estimators, but the usual conditions are the so-called "Grenander conditions" (see e.g., Grenander 1954), which we will discuss below.

---

OLS consistency does not follow from the ordinary regression model assumptions: Without imposing any conditions here, there is no guarantee that the variance of the OLS estimators will reduce to zero asymptotically, even under the first of your two equations. (As an example of this, consider the case where the sequence of $x_i$ values converges rapidly to its mean so that it stops adding anything more to the sum-of-squares after a finite number of observations.) In the general case, the OLS estimator can be written in the form:

$$\hat{\boldsymbol{\beta}} = \boldsymbol{\beta} + \Bigg( \frac{1}{n} \ \mathbf{x}^\text{T} \mathbf{x} \Bigg)^{-1} \Bigg( \frac{1}{n} \ \mathbf{x}^\text{T} \boldsymbol{\varepsilon} \Bigg),$$

so we have the probability limit:

$$\text{plim} \ \hat{\boldsymbol{\beta}} = \boldsymbol{\beta} + \Bigg( \text{plim} \ \frac{1}{n} \ \mathbf{x}^\text{T} \mathbf{x} \Bigg)^{-1} \Bigg( \text{plim} \ \frac{1}{n} \ \mathbf{x}^\text{T} \boldsymbol{\varepsilon} \Bigg).$$

The standard regression assumptions give IID error terms with $\mathbb{E}(\boldsymbol{\varepsilon}|\mathbf{x}) = \mathbf{0}$. Using the laws of iterated expectation and variance, the moments of the second quantity are:

$$\mathbb{E} \Big( \frac{1}{n} \ \mathbf{x}^\text{T} \boldsymbol{\varepsilon} \Big) = \mathbf{0} \quad \quad \quad \mathbb{V} \Big( \frac{1}{n} \ \mathbf{x}^\text{T} \boldsymbol{\varepsilon} \Big) = \frac{\sigma^2}{n} \cdot \frac{\mathbf{x}^\text{T} \mathbf{x}}{n}.$$

The OLS estimator is consistent if and only if $\text{plim} \ \hat{\boldsymbol{\beta}} = \boldsymbol{\beta}$, which occurs when the second term in the above expression vanishes (i.e., is equal to the zero vector). The first thing to observe is that there is no guarantee that this occurs under the standard regression model assumptions. If the asymptotic behavious of the explanatory variables is nasty, then the $\text{plim}$ in the first brackets might be zero, or the $\text{plim}$ in the second brackets might not be zero. In the example I mentioned above, where the explanatory variables rapidly converge to their mean, the $\text{plim}$ in the first brackets will be zero, and the inverted term explodes.

---

OLS consistency comes from the limiting behaviour of the explanatory variables To obtain asymptotic consistency of the OLS estimator, it is sufficient to show that $\mathbb{V}(\tfrac{1}{n} \ \mathbf{x}^\text{T} \boldsymbol{\varepsilon}) \rightarrow 0$ (so that the second $\text{plim}$ converges to the zero vector) and that the first bracketed term converges to a fixed matrix. This is where we use the "Grenander conditions", or some other sufficient conditions. In particular, it is sufficient here to assume (or establish from other assumptions) that there exists some probability limit $\text{plim} \ \tfrac{1}{n} \ \mathbf{x}^\text{T} \mathbf{x} = \mathbf{Q}$ where $\mathbf{Q}$ is a positive definite matrix with non-zero diagonal elements. This is a sufficient condition for asymptotic consistency, since it ensure that the first $\text{plim}$ in the above expression is non-zero and the second is zero. This ensures that the entire term vanishes, leaving the consistency property $\text{plim} \ \hat{\boldsymbol{\beta}} = \boldsymbol{\beta}$.