3
$\begingroup$

In the first answer to this post, it is shown that the variance of the estimated regression coefficient $\hat \beta_1$ in simple linear regression is $$ \text{Var}(\hat \beta_1) = \frac{\sigma^2}{\sum_i (x_i - \bar x)^2}. \quad \quad \quad (1) $$

On Wikipedia we have the following expression for the variance-covariance matrix of the vector of coefficients: $$ \text{Var}(\hat \beta|X) = \sigma^2 (X^T X)^{-1}. \quad \quad \quad (2) $$

The expressions (1) and (2) are quite similar. In expression (1), it is clear that for larger samples sizes, the variance will decrease as we will have more terms in the summation in the denominator. But for expression 2, it seems that there is no guarantee that the variance (the diagonals of the covariance matrix) will decrease if we take larger samples sizes?

Also, can expression (1) be obtained from expression (2)?

Edit: The result for $\text{Var}(\hat \beta_1$) in (1) should also be contained in the a corresponding entry on the diagonal of $\text{Var}(\hat \beta | X)$? How do we show that (1) corresponds to an entry on the diagonal of (2)? I'm not seeing where the subtraction of the mean in the denominator of (1) arises in (2)?

$\endgroup$

1 Answer 1

5
$\begingroup$

In regression analysis, if you want to get asymptotic consistency results you need to impose some limiting conditions on the explanatory variables. The regression model itself makes no assumptions about the form of the sequence of explanatory variables, so these limiting conditions are conditions that go beyond the model assumptions for regression analysis. There are various (sufficient) limiting conditions that can be imposed on the sequence of explanatory variables in order to get consistency of the OLS estimators, but the usual conditions are the so-called "Grenander conditions" (see e.g., Grenander 1954), which we will discuss below.


OLS consistency does not follow from the ordinary regression model assumptions: Without imposing any conditions here, there is no guarantee that the variance of the OLS estimators will reduce to zero asymptotically, even under the first of your two equations. (As an example of this, consider the case where the sequence of $x_i$ values converges rapidly to its mean so that it stops adding anything more to the sum-of-squares after a finite number of observations.) In the general case, the OLS estimator can be written in the form:

$$\hat{\boldsymbol{\beta}} = \boldsymbol{\beta} + \Bigg( \frac{1}{n} \ \mathbf{x}^\text{T} \mathbf{x} \Bigg)^{-1} \Bigg( \frac{1}{n} \ \mathbf{x}^\text{T} \boldsymbol{\varepsilon} \Bigg),$$

so we have the probability limit:

$$\text{plim} \ \hat{\boldsymbol{\beta}} = \boldsymbol{\beta} + \Bigg( \text{plim} \ \frac{1}{n} \ \mathbf{x}^\text{T} \mathbf{x} \Bigg)^{-1} \Bigg( \text{plim} \ \frac{1}{n} \ \mathbf{x}^\text{T} \boldsymbol{\varepsilon} \Bigg).$$

The standard regression assumptions give IID error terms with $\mathbb{E}(\boldsymbol{\varepsilon}|\mathbf{x}) = \mathbf{0}$. Using the laws of iterated expectation and variance, the moments of the second quantity are:

$$\mathbb{E} \Big( \frac{1}{n} \ \mathbf{x}^\text{T} \boldsymbol{\varepsilon} \Big) = \mathbf{0} \quad \quad \quad \mathbb{V} \Big( \frac{1}{n} \ \mathbf{x}^\text{T} \boldsymbol{\varepsilon} \Big) = \frac{\sigma^2}{n} \cdot \frac{\mathbf{x}^\text{T} \mathbf{x}}{n}.$$

The OLS estimator is consistent if and only if $\text{plim} \ \hat{\boldsymbol{\beta}} = \boldsymbol{\beta}$, which occurs when the second term in the above expression vanishes (i.e., is equal to the zero vector). The first thing to observe is that there is no guarantee that this occurs under the standard regression model assumptions. If the asymptotic behavious of the explanatory variables is nasty, then the $\text{plim}$ in the first brackets might be zero, or the $\text{plim}$ in the second brackets might not be zero. In the example I mentioned above, where the explanatory variables rapidly converge to their mean, the $\text{plim}$ in the first brackets will be zero, and the inverted term explodes.


OLS consistency comes from the limiting behaviour of the explanatory variables To obtain asymptotic consistency of the OLS estimator, it is sufficient to show that $\mathbb{V}(\tfrac{1}{n} \ \mathbf{x}^\text{T} \boldsymbol{\varepsilon}) \rightarrow 0$ (so that the second $\text{plim}$ converges to the zero vector) and that the first bracketed term converges to a fixed matrix. This is where we use the "Grenander conditions", or some other sufficient conditions. In particular, it is sufficient here to assume (or establish from other assumptions) that there exists some probability limit $\text{plim} \ \tfrac{1}{n} \ \mathbf{x}^\text{T} \mathbf{x} = \mathbf{Q}$ where $\mathbf{Q}$ is a positive definite matrix with non-zero diagonal elements. This is a sufficient condition for asymptotic consistency, since it ensure that the first $\text{plim}$ in the above expression is non-zero and the second is zero. This ensures that the entire term vanishes, leaving the consistency property $\text{plim} \ \hat{\boldsymbol{\beta}} = \boldsymbol{\beta}$.

$\endgroup$
10
  • 1
    $\begingroup$ Your (counter) example is interesting as it shows that I have made a mistake in my other recent post - stats.stackexchange.com/questions/491005/… $\endgroup$ Commented Oct 8, 2020 at 12:04
  • 1
    $\begingroup$ The rate of convergence would depend on the properties of the sequence of explanatory variables, so if you wanted results pertaining to this you would need to stipulate further conditions on these variables. $\endgroup$
    – Ben
    Commented Oct 9, 2020 at 0:16
  • 1
    $\begingroup$ Yes, but remember that that is an assumption about the explanatory variables, not something that follows from the initial assumptions of the regression model. $\endgroup$
    – Ben
    Commented Oct 9, 2020 at 13:05
  • 1
    $\begingroup$ Again, that quantity depends on $\mathbf{x}$. So you tell me --- is it? $\endgroup$
    – Ben
    Commented Oct 13, 2020 at 21:29
  • 1
    $\begingroup$ If you want to get an "order" of convergence then you need to make some assumption about the corresponding "order" of the explanatory variables. $\endgroup$
    – Ben
    Commented Oct 26, 2021 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.