Consider the simple linear regression model. $$ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \quad \quad \quad \quad i = 1,2,\dots,n. $$ Let $\mu_x$ and $\sigma_x^2$ represent the mean and variance of the i.i.d. observations $x_1,x_2,\dots,x_n$. The simple linear regression model can be written in matrix form as $$ y = \mathbf{X} \beta + \varepsilon, $$ where $$ \mathbf{X} = \begin{bmatrix} \mathbf{x_1}^T \\ \mathbf{x_2}^T \\ \vdots \\ \mathbf{x_n}^T \end{bmatrix} = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix}. $$

Regarding the large sample theory for regression, it is known that $$ \bigg(\frac{\mathbf{X}^T \mathbf{X}}{n}\bigg) \stackrel{p}{\longrightarrow} E[\mathbf{x}_i \mathbf{x}_i']. $$ This is shown in equation (6.1) in these notes, where the author uses the notation \begin{align} \hat Q_{xx} = \bigg(\frac{\mathbf{X}^T \mathbf{X}}{n}\bigg), \\ Q_{xx} = E[\mathbf{x}_i \mathbf{x}_i']. \end{align}

Sticking with the notation in those notes, we then have $$ \hat Q_{xx}^{-1} \stackrel{p}{\longrightarrow} Q_{xx}^{-1}. $$ I want to find the actual rate of convergence in probability of $\hat Q_{xx}^{-1}$ to $Q_{xx}^{-1}$. The above expression means that $\hat Q_{xx}^{-1} - Q_{xx}^{-1} = o_p(1)$, but I am hoping that the convergence can be shown more precisely or by shown to be faster, possibly something like $\hat Q_{xx}^{-1} - Q_{xx}^{-1} = O_p(n^{-1/2})$ or $o_p(n^{-1/2})$.

Since we are considering simple linear regression we have an explicit representation for $\hat Q_{xx}^{-1}$: $$ \begin{align} \hat Q_{xx}^{-1} = n (\mathbf{X}^T \mathbf{X})^{-1} & = \frac{1}{\sum_{i=1}^n(x_i - \overline x)^2} \begin{bmatrix} \sum_{i=1}^n x_i^2 & - \sum_{i=1}^n x_i \\ - \sum_{i=1}^n x_i & n \end{bmatrix}, \\ & = \frac{1}{\frac{1}{n}\sum_{i=1}^n(x_i - \overline x)^2} \begin{bmatrix} \frac{1}{n} \sum_{i=1}^n x_i^2 & - \frac{1}{n} \sum_{i=1}^n x_i \\ - \frac{1}{n} \sum_{i=1}^n x_i & \frac{1}{n} n \end{bmatrix}, \\ & = \frac{1}{s^2} \begin{bmatrix} \gamma & - \overline x \\ - \overline x & 1 \end{bmatrix}. \end{align} $$

Then we have:

  1. $\sqrt{n}(s^2 - \sigma_x^2) \stackrel{d}{\longrightarrow} \mathcal{N}(0,E[x_i - \mu_x^4])$ (shown here)
  2. $\sqrt{n}(\overline x - \mu_x) \stackrel{d}{\longrightarrow} \mathcal{N}(0,\sigma_x^2)$
  3. $\sqrt{n}(\gamma - (?)) \stackrel{d}{\longrightarrow} \mathcal{N}(0,(?))$ (I'm not sure about this one)

In any, case the numerator and denominator of each element of the $2\times 2$ matrix $\hat Q_{xx}^{-1}$ is converging in distribution, and since they are converging to a constant, we get convergence in probability. Due to the $\sqrt{n}$ term at the front of each expression it appears we can say that the convergence of $\hat Q_{xx}^{-1}$ to $Q_{xx}^{-1}$ really is faster than $o_p(1)$. It appears we can say $$ \hat Q_{xx}^{-1} - Q_{xx}^{-1} = O_p(n^{-1/2}). $$

Is this in fact the case, do we really have a rate of convergence of $O_p(n^{-1/2})$ or have I made a mistake?


1 Answer 1


It would be easier to use block-matrix notation, but it's ok.

Under the implied regularity assumptions that you have an i.i.d. sample, hence ergodic, and that the various expected values exist, your "$\gamma$", let's denote it $\gamma_n$, converges to $E(x^2)$.

You obtained

$$\hat Q_{xx}^{-1} - Q_{xx}^{-1} = \frac{1}{s^2} \begin{bmatrix} \gamma_n & - \overline x \\ - \overline x & 1 \end{bmatrix} \;-\;\frac{1}{\sigma^2_x} \begin{bmatrix} E(x^2) & - \mu_x\\ - \mu_x & 1 \end{bmatrix}$$

$$\implies \sqrt{n}\left(\hat Q_{xx}^{-1} - Q_{xx}^{-1}\right) = \begin{bmatrix} \sqrt{n}\left(\frac{\gamma_n}{s^2}-\frac{E(x^2)}{\sigma^2_x}\right) & -\sqrt{n}\left(\frac{\overline x}{s^2}-\frac{\mu_x}{\sigma^2_x}\right) \\ -\sqrt{n}\left(\frac{\overline x}{s^2}-\frac{\mu_x}{\sigma^2_x}\right) & \sqrt{n}\left(\frac{1}{s^2}-\frac{1}{\sigma^2_x}\right) \end{bmatrix}$$

Now you have to show that each element of this last matrix converges in distribution to a random variable, or to a constant. In such a case you will get that

$$\hat Q_{xx}^{-1} - Q_{xx}^{-1} = O_p(n^{-1/2}),$$

and so that it is $o_p(n^{-\delta}),\; 0 \leq \delta < 1/2$, and therefore that convergence is faster than just $o_p(1)$.

For the upper row (and so the left lower element), this is true using the i.i.d. sample assumption, Slutsky's Lemma and the symmetry of the Normal distribution around its mean (they obey a classical CLT). A little care with the lower right element.

See this post, https://stats.stackexchange.com/a/379971/28746 for when we can go from convergence in distribution to convergence in probability (in fact to the even stronger convergence in quadratic mean).

  • $\begingroup$ Ok great, thanks for the clarification. Just one thing I noticed is that we will use the CLT to show that the elements of the matrix converge in distribution to mean zero normal distributions with different variances. e.g. $\sqrt{n}\left(\frac{\overline x}{s^2}-\frac{\mu_x}{\sigma^2_x}\right) \stackrel{d}{\to} \mathcal{N}(0,c)$ for $c>0$. But then how can we get from this to $\sqrt{n}\left(\frac{\overline x}{s^2}-\frac{\mu_x}{\sigma^2_x}\right) \stackrel{p}{\to} 0$ since convergence in distribution only implies convergence in probability if the convergence is to a constant? $\endgroup$
    – sonicboom
    Commented Dec 12, 2020 at 18:44
  • $\begingroup$ On a related note, Hansen has $\hat \beta = \beta + O_p(n^{-1/2})$ in (6.13) since $\sqrt{n}(\hat \beta - \beta) \stackrel{d}{\to} \mathcal{N}(0,V)$ so he also upgrades the convergence in the same fashion, I just don't see what result is being used to upgrade from convergence in distribution to convergence in probability? $\endgroup$
    – sonicboom
    Commented Dec 12, 2020 at 18:54
  • 1
    $\begingroup$ @sonicboom See the addition I made to the post. $\endgroup$ Commented Dec 12, 2020 at 21:29
  • $\begingroup$ Very nice, I see how the process works now. $\endgroup$
    – sonicboom
    Commented Dec 12, 2020 at 23:24
  • 1
    $\begingroup$ @sonicboom. You do not need to prove convergence in probability through convergence in distribution. Convergence in probability is proved on its own for the $Q^{-1}$ matrix. Proving the convergence in distribution is done to be able to improve on the rate of convergence, not to prove convergence in probability. $\endgroup$ Commented Dec 13, 2020 at 0:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.