Consider the linear model $$y_i = x_i^T\beta + \epsilon_i.$$

In ordinary least squares it is assumed that the errors satisfy $E[\epsilon_i]=0$. This implies that that $\dfrac{X^T\epsilon}{n} \to 0$ in probability, where $X$ is the design matrix and it is non-stochastic. What is the reasoning behind this claim?


That result is not implied by the distribution of the error terms. Asymptotic convergence results in regression analysis ---which treat the explanatory variables as fixed--- require a separate assumption that ensure that the explanatory variables do not grow too rapidly in magnitude. In the case of IID error terms with $\mathbb{E}(\epsilon_i) = 0$ and $\mathbb{V}(\epsilon_i) = \sigma^2$ you have:

$$\bigg( \frac{\mathbf{x}_n \cdot \boldsymbol{\epsilon}_n}{n} \bigg)^2 \leqslant \frac{||\mathbf{x}_n||^2 \cdot ||\boldsymbol{\epsilon_n}||^2}{n^2} = \frac{||\mathbf{x}_n||^2}{n} \cdot \frac{||\boldsymbol{\epsilon}_n||^2}{n} \rightarrow \frac{\sum_{i=1}^n x_i^2}{n} \cdot \sigma^2.$$

Thus, a sufficient condition for the convergence you are seeking is to assume that $\sum_{i=1}^n x_i^2/n \rightarrow 0$.

  • $\begingroup$ This condition appears to be unnecessarily strong. $\endgroup$ – Alecos Papadopoulos Feb 14 at 2:51

We essentially examine the conditions under which the Law of Large Numbers holds for the sum

$$\frac 1n \sum_{i=1}^nx_{ki}u_i,\;\;\; E(u_i)=0$$

for every $k=1,...K$ regressor, and we assume also a finite variance for $u_i$.

Now, when the $x_{ki}$'s are non-stochastic then they are just a sequence of real numbers, and we might as well re-write the sum as

$$\frac 1n \sum_{i=1}^na_{ki}u_i,\;\;\; E(u_i)=0$$

I write it like this to stress that the only source of randomness here is the $u_i$'s, and so that, what we are looking at is the average of independent but non-identically distributed random variables $z_{ki}=a_{ki}u_i$. This is the "Chebyshev's" Law of Large Numbers and requires that the variance of each random variable is finite. This means that we need $$\text{Var}(z_{ki}) = a_{ki}^2 \text{Var}(u_i) < \infty,\;\;\; \forall i$$

Markov generalized this LLN to possibly non-independent random variables, where we require that

$$\text{Var}\left(\frac 1n \sum_{i=1}^na_{ki}u_i\right) \to 0$$

Under independence of $u_i$'s, covariances are zero and we have

$$\text{Var}\left(\frac 1n \sum_{i=1}^na_{ki}u_i\right) = \text{Var}(u_i)\cdot \frac 1{n^2}\sum_{i=1}^na_{ki}^2$$ and so the sufficient condition we need is

$$\frac 1{n^2}\sum_{i=1}^na_{ki}^2 \to 0$$

or to revert back to original notation

$$\frac 1{n^2}\sum_{i=1}^nx_{ki}^2 \to 0$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.